1
|
Cui H, Li Y, Wang Y, Xu D, Wu LM, Xia Y. Toward Accurate Cardiac MRI Segmentation With Variational Autoencoder-Based Unsupervised Domain Adaptation. IEEE TRANSACTIONS ON MEDICAL IMAGING 2024; 43:2924-2936. [PMID: 38546999 DOI: 10.1109/tmi.2024.3382624] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 08/02/2024]
Abstract
Accurate myocardial segmentation is crucial in the diagnosis and treatment of myocardial infarction (MI), especially in Late Gadolinium Enhancement (LGE) cardiac magnetic resonance (CMR) images, where the infarcted myocardium exhibits a greater brightness. However, segmentation annotations for LGE images are usually not available. Although knowledge gained from CMR images of other modalities with ample annotations, such as balanced-Steady State Free Precession (bSSFP), can be transferred to the LGE images, the difference in image distribution between the two modalities (i.e., domain shift) usually results in a significant degradation in model performance. To alleviate this, an end-to-end Variational autoencoder based feature Alignment Module Combining Explicit and Implicit features (VAMCEI) is proposed. We first re-derive the Kullback-Leibler (KL) divergence between the posterior distributions of the two domains as a measure of the global distribution distance. Second, we calculate the prototype contrastive loss between the two domains, bringing closer the prototypes of the same category across domains and pushing away the prototypes of different categories within or across domains. Finally, a domain discriminator is added to the output space, which indirectly aligns the feature distribution and forces the extracted features to be more favorable for segmentation. In addition, by combining CycleGAN and VAMCEI, we propose a more refined multi-stage unsupervised domain adaptation (UDA) framework for myocardial structure segmentation. We conduct extensive experiments on the MSCMRSeg 2019, MyoPS 2020 and MM-WHS 2017 datasets. The experimental results demonstrate that our framework achieves superior performances than state-of-the-art methods.
Collapse
|
2
|
Ganesan P, Feng R, Deb B, Tjong FVY, Rogers AJ, Ruipérez-Campillo S, Somani S, Clopton P, Baykaner T, Rodrigo M, Zou J, Haddad F, Zaharia M, Narayan SM. Novel Domain Knowledge-Encoding Algorithm Enables Label-Efficient Deep Learning for Cardiac CT Segmentation to Guide Atrial Fibrillation Treatment in a Pilot Dataset. Diagnostics (Basel) 2024; 14:1538. [PMID: 39061675 PMCID: PMC11276420 DOI: 10.3390/diagnostics14141538] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/15/2024] [Revised: 07/07/2024] [Accepted: 07/10/2024] [Indexed: 07/28/2024] Open
Abstract
Background: Segmenting computed tomography (CT) is crucial in various clinical applications, such as tailoring personalized cardiac ablation for managing cardiac arrhythmias. Automating segmentation through machine learning (ML) is hindered by the necessity for large, labeled training data, which can be challenging to obtain. This article proposes a novel approach for automated, robust labeling using domain knowledge to achieve high-performance segmentation by ML from a small training set. The approach, the domain knowledge-encoding (DOKEN) algorithm, reduces the reliance on large training datasets by encoding cardiac geometry while automatically labeling the training set. The method was validated in a hold-out dataset of CT results from an atrial fibrillation (AF) ablation study. Methods: The DOKEN algorithm parses left atrial (LA) structures, extracts "anatomical knowledge" by leveraging digital LA models (available publicly), and then applies this knowledge to achieve high ML segmentation performance with a small number of training samples. The DOKEN-labeled training set was used to train a nnU-Net deep neural network (DNN) model for segmenting cardiac CT in N = 20 patients. Subsequently, the method was tested in a hold-out set with N = 100 patients (five times larger than training set) who underwent AF ablation. Results: The DOKEN algorithm integrated with the nn-Unet model achieved high segmentation performance with few training samples, with a training to test ratio of 1:5. The Dice score of the DOKEN-enhanced model was 96.7% (IQR: 95.3% to 97.7%), with a median error in surface distance of boundaries of 1.51 mm (IQR: 0.72 to 3.12) and a mean centroid-boundary distance of 1.16 mm (95% CI: -4.57 to 6.89), similar to expert results (r = 0.99; p < 0.001). In digital hearts, the novel DOKEN approach segmented the LA structures with a mean difference for the centroid-boundary distances of -0.27 mm (95% CI: -3.87 to 3.33; r = 0.99; p < 0.0001). Conclusions: The proposed novel domain knowledge-encoding algorithm was able to perform the segmentation of six substructures of the LA, reducing the need for large training data sets. The combination of domain knowledge encoding and a machine learning approach could reduce the dependence of ML on large training datasets and could potentially be applied to AF ablation procedures and extended in the future to other imaging, 3D printing, and data science applications.
Collapse
Affiliation(s)
- Prasanth Ganesan
- Department of Medicine and Stanford Cardiovascular Institute (CVI), Stanford University, Stanford, CA 94305, USA; (P.G.); (R.F.)
| | - Ruibin Feng
- Department of Medicine and Stanford Cardiovascular Institute (CVI), Stanford University, Stanford, CA 94305, USA; (P.G.); (R.F.)
| | - Brototo Deb
- Department of Medicine and Stanford Cardiovascular Institute (CVI), Stanford University, Stanford, CA 94305, USA; (P.G.); (R.F.)
| | - Fleur V. Y. Tjong
- Department of Medicine and Stanford Cardiovascular Institute (CVI), Stanford University, Stanford, CA 94305, USA; (P.G.); (R.F.)
- Heart Center, Department of Clinical and Experimental Cardiology, Amsterdam UMC, University of Amsterdam, 1105 AZ Amsterdam, The Netherlands
| | - Albert J. Rogers
- Department of Medicine and Stanford Cardiovascular Institute (CVI), Stanford University, Stanford, CA 94305, USA; (P.G.); (R.F.)
| | - Samuel Ruipérez-Campillo
- Department of Medicine and Stanford Cardiovascular Institute (CVI), Stanford University, Stanford, CA 94305, USA; (P.G.); (R.F.)
- Department of Computer Science, ETH Zurich, 8092 Zurich, Switzerland
| | - Sulaiman Somani
- Department of Medicine and Stanford Cardiovascular Institute (CVI), Stanford University, Stanford, CA 94305, USA; (P.G.); (R.F.)
| | - Paul Clopton
- Department of Medicine and Stanford Cardiovascular Institute (CVI), Stanford University, Stanford, CA 94305, USA; (P.G.); (R.F.)
| | - Tina Baykaner
- Department of Medicine and Stanford Cardiovascular Institute (CVI), Stanford University, Stanford, CA 94305, USA; (P.G.); (R.F.)
| | - Miguel Rodrigo
- Department of Medicine and Stanford Cardiovascular Institute (CVI), Stanford University, Stanford, CA 94305, USA; (P.G.); (R.F.)
- CoMMLab, Universitat de València, 46100 Valencia, Spain
| | - James Zou
- Department of Biomedical Data Science, Stanford University, Stanford, CA 94305, USA
| | - Francois Haddad
- Department of Medicine and Stanford Cardiovascular Institute (CVI), Stanford University, Stanford, CA 94305, USA; (P.G.); (R.F.)
| | - Matei Zaharia
- Department of Computer Science, University of California Berkeley, Berkeley, CA 94720, USA
| | - Sanjiv M. Narayan
- Department of Medicine and Stanford Cardiovascular Institute (CVI), Stanford University, Stanford, CA 94305, USA; (P.G.); (R.F.)
| |
Collapse
|
3
|
Zwijnen AW, Watzema L, Ridwan Y, van Der Pluijm I, Smal I, Essers J. Self-adaptive deep learning-based segmentation for universal and functional clinical and preclinical CT image analysis. Comput Biol Med 2024; 179:108853. [PMID: 39013341 DOI: 10.1016/j.compbiomed.2024.108853] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/01/2023] [Revised: 07/04/2024] [Accepted: 07/04/2024] [Indexed: 07/18/2024]
Abstract
BACKGROUND Methods to monitor cardiac functioning non-invasively can accelerate preclinical and clinical research into novel treatment options for heart failure. However, manual image analysis of cardiac substructures is resource-intensive and error-prone. While automated methods exist for clinical CT images, translating these to preclinical μCT data is challenging. We employed deep learning to automate the extraction of quantitative data from both CT and μCT images. METHODS We collected a public dataset of cardiac CT images of human patients, as well as acquired μCT images of wild-type and accelerated aging mice. The left ventricle, myocardium, and right ventricle were manually segmented in the μCT training set. After template-based heart detection, two separate segmentation neural networks were trained using the nnU-Net framework. RESULTS The mean Dice score of the CT segmentation results (0.925 ± 0.019, n = 40) was superior to those achieved by state-of-the-art algorithms. Automated and manual segmentations of the μCT training set were nearly identical. The estimated median Dice score (0.940) of the test set results was comparable to existing methods. The automated volume metrics were similar to manual expert observations. In aging mice, ejection fractions had significantly decreased, and myocardial volume increased by age 24 weeks. CONCLUSIONS With further optimization, automated data extraction expands the application of (μ)CT imaging, while reducing subjectivity and workload. The proposed method efficiently measures the left and right ventricular ejection fraction and myocardial mass. With uniform translation between image types, cardiac functioning in diastolic and systolic phases can be monitored in both animals and humans.
Collapse
Affiliation(s)
- Anne-Wietje Zwijnen
- Department of Molecular Genetics, Erasmus University Medical Center, Rotterdam, the Netherlands
| | | | - Yanto Ridwan
- AMIE Core Facility, Erasmus Medical Center, Rotterdam, the Netherlands
| | - Ingrid van Der Pluijm
- Department of Molecular Genetics, Erasmus University Medical Center, Rotterdam, the Netherlands; Department of Vascular Surgery, Erasmus University Medical Center, Rotterdam, the Netherlands
| | - Ihor Smal
- Department of Cell Biology, Erasmus University Medical Center, Rotterdam, the Netherlands
| | - Jeroen Essers
- Department of Molecular Genetics, Erasmus University Medical Center, Rotterdam, the Netherlands; Department of Vascular Surgery, Erasmus University Medical Center, Rotterdam, the Netherlands; Department of Radiotherapy, Erasmus University Medical Center, Rotterdam, the Netherlands.
| |
Collapse
|
4
|
Pace DF, Contreras HTM, Romanowicz J, Ghelani S, Rahaman I, Zhang Y, Gao P, Jubair MI, Yeh T, Golland P, Geva T, Ghelani S, Powell AJ, Moghari MH. HVSMR-2.0: A 3D cardiovascular MR dataset for whole-heart segmentation in congenital heart disease. Sci Data 2024; 11:721. [PMID: 38956063 PMCID: PMC11219801 DOI: 10.1038/s41597-024-03469-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/11/2023] [Accepted: 06/04/2024] [Indexed: 07/04/2024] Open
Abstract
Patients with congenital heart disease often have cardiac anatomy that deviates significantly from normal, frequently requiring multiple heart surgeries. Image segmentation from a preoperative cardiovascular magnetic resonance (CMR) scan would enable creation of patient-specific 3D surface models of the heart, which have potential to improve surgical planning, enable surgical simulation, and allow automatic computation of quantitative metrics of heart function. However, there is no publicly available CMR dataset for whole-heart segmentation in patients with congenital heart disease. Here, we release the HVSMR-2.0 dataset, comprising 60 CMR scans alongside manual segmentation masks of the 4 cardiac chambers and 4 great vessels. The images showcase a wide range of heart defects and prior surgical interventions. The dataset also includes masks of required and optional extents of the great vessels, enabling fairer comparisons across algorithms. Detailed diagnoses for each subject are also provided. By releasing HVSMR-2.0, we aim to encourage development of robust segmentation algorithms and clinically relevant tools for congenital heart disease.
Collapse
Affiliation(s)
- Danielle F Pace
- A.A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital, Boston, MA, USA.
- Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, MA, USA.
- Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology, Cambridge, MA, USA.
| | - Hannah T M Contreras
- Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, MA, USA
- Pediatric Surgical Research Laboratories, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
| | - Jennifer Romanowicz
- Department of Pediatrics, Section of Cardiology, Children's Hospital Colorado, Aurora, CO, USA
| | - Shruti Ghelani
- Department of Computer Science, University of Massachusetts Boston, Boston, MA, USA
| | - Imon Rahaman
- Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, MA, USA
- Department of Biological Engineering, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Yue Zhang
- Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, MA, USA
- Feinberg School of Medicine, Northwestern University, Chicago, IL, USA
- Department of Biochemistry and Molecular Genetics, Northwestern University, Chicago, IL, USA
- School of Medicine, Indiana University, Indianapolis, IN, USA
| | - Patricia Gao
- Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, MA, USA
- Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology, Cambridge, MA, USA
| | | | - Tom Yeh
- Department of Computer Science, University of Colorado Boulder, Boulder, CO, USA
- Department of Psychology, Ewha Womans University, Seoul, South Korea
| | - Polina Golland
- Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, MA, USA
- Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Tal Geva
- Department of Cardiology, Boston Children's Hospital, Boston, MA, USA
- Department of Pediatrics, Harvard Medical School, Boston, MA, USA
| | - Sunil Ghelani
- Department of Cardiology, Boston Children's Hospital, Boston, MA, USA
- Department of Pediatrics, Harvard Medical School, Boston, MA, USA
| | - Andrew J Powell
- Department of Cardiology, Boston Children's Hospital, Boston, MA, USA
- Department of Pediatrics, Harvard Medical School, Boston, MA, USA
| | - Mehdi Hedjazi Moghari
- Department of Cardiology, Boston Children's Hospital, Boston, MA, USA
- Department of Pediatrics, Harvard Medical School, Boston, MA, USA
- School of Medicine, The University of Colorado, Aurora, CO, USA
- Department of Radiology, Children's Hospital Colorado, Aurora, CO, USA
| |
Collapse
|
5
|
Narayanan A, Kong F, Shadden S. LinFlo-Net: A Two-Stage Deep Learning Method to Generate Simulation Ready Meshes of the Heart. J Biomech Eng 2024; 146:071005. [PMID: 38258957 DOI: 10.1115/1.4064527] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/16/2023] [Accepted: 01/19/2024] [Indexed: 01/24/2024]
Abstract
We present a deep learning model to automatically generate computer models of the human heart from patient imaging data with an emphasis on its capability to generate thin-walled cardiac structures. Our method works by deforming a template mesh to fit the cardiac structures to the given image. Compared with prior deep learning methods that adopted this approach, our framework is designed to minimize mesh self-penetration, which typically arises when deforming surface meshes separated by small distances. We achieve this by using a two-stage diffeomorphic deformation process along with a novel loss function derived from the kinematics of motion that penalizes surface contact and interpenetration. Our model demonstrates comparable accuracy with state-of-the-art methods while additionally producing meshes free of self-intersections. The resultant meshes are readily usable in physics based simulation, minimizing the need for postprocessing and cleanup.
Collapse
Affiliation(s)
- Arjun Narayanan
- Department of Mechanical Engineering, University of California, Berkeley, Berkeley, CA 94709
| | - Fanwei Kong
- Department of Pediatrics, Stanford University, Stanford, CA 94305
- Stanford University
| | - Shawn Shadden
- Department of Mechanical Engineering, University of California, Berkeley, Berkeley, CA 94709
| |
Collapse
|
6
|
Melkani Y, Pant A, Guo Y, Melkani GC. Automated assessment of cardiac dynamics in aging and dilated cardiomyopathy Drosophila models using machine learning. Commun Biol 2024; 7:702. [PMID: 38849449 PMCID: PMC11161577 DOI: 10.1038/s42003-024-06371-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/18/2024] [Accepted: 05/22/2024] [Indexed: 06/09/2024] Open
Abstract
The Drosophila model is pivotal in deciphering the pathophysiological underpinnings of various human ailments, notably aging and cardiovascular diseases. Cutting-edge imaging techniques and physiology yield vast high-resolution videos, demanding advanced analysis methods. Our platform leverages deep learning to segment optical microscopy images of Drosophila hearts, enabling the quantification of cardiac parameters in aging and dilated cardiomyopathy (DCM). Validation using experimental datasets confirms the efficacy of our aging model. We employ two innovative approaches deep-learning video classification and machine-learning based on cardiac parameters to predict fly aging, achieving accuracies of 83.3% (AUC 0.90) and 79.1%, (AUC 0.87) respectively. Moreover, we extend our deep-learning methodology to assess cardiac dysfunction associated with the knock-down of oxoglutarate dehydrogenase (OGDH), revealing its potential in studying DCM. This versatile approach promises accelerated cardiac assays for modeling various human diseases in Drosophila and holds promise for application in animal and human cardiac physiology under diverse conditions.
Collapse
Affiliation(s)
- Yash Melkani
- Department of Pathology, Division of Molecular and Cellular Pathology, Heersink School of Medicine, University of Alabama at Birmingham, Birmingham, AL, USA
- Engineering Physics Department, College of Engineering, University of California, Berkeley, CA, USA
| | - Aniket Pant
- Department of Pathology, Division of Molecular and Cellular Pathology, Heersink School of Medicine, University of Alabama at Birmingham, Birmingham, AL, USA
- Department of Materials Science and Engineering, Georgia Institute of Technology, Atlanta, GA, USA
| | - Yiming Guo
- Department of Pathology, Division of Molecular and Cellular Pathology, Heersink School of Medicine, University of Alabama at Birmingham, Birmingham, AL, USA
| | - Girish C Melkani
- Department of Pathology, Division of Molecular and Cellular Pathology, Heersink School of Medicine, University of Alabama at Birmingham, Birmingham, AL, USA.
| |
Collapse
|
7
|
Chen B, Li Y, Liu J, Yang F, Zhang L. MSMHSA-DeepLab V3+: An Effective Multi-Scale, Multi-Head Self-Attention Network for Dual-Modality Cardiac Medical Image Segmentation. J Imaging 2024; 10:135. [PMID: 38921612 PMCID: PMC11204943 DOI: 10.3390/jimaging10060135] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/29/2024] [Revised: 05/29/2024] [Accepted: 05/31/2024] [Indexed: 06/27/2024] Open
Abstract
The automatic segmentation of cardiac computed tomography (CT) and magnetic resonance imaging (MRI) plays a pivotal role in the prevention and treatment of cardiovascular diseases. In this study, we propose an efficient network based on the multi-scale, multi-head self-attention (MSMHSA) mechanism. The incorporation of this mechanism enables us to achieve larger receptive fields, facilitating the accurate segmentation of whole heart structures in both CT and MRI images. Within this network, features extracted from the shallow feature extraction network undergo a MHSA mechanism that closely aligns with human vision, resulting in the extraction of contextual semantic information more comprehensively and accurately. To improve the precision of cardiac substructure segmentation across varying sizes, our proposed method introduces three MHSA networks at distinct scales. This approach allows for fine-tuning the accuracy of micro-object segmentation by adapting the size of the segmented images. The efficacy of our method is rigorously validated on the Multi-Modality Whole Heart Segmentation (MM-WHS) Challenge 2017 dataset, demonstrating competitive results and the accurate segmentation of seven cardiac substructures in both cardiac CT and MRI images. Through comparative experiments with advanced transformer-based models, our study provides compelling evidence that despite the remarkable achievements of transformer-based models, the fusion of CNN models and self-attention remains a simple yet highly effective approach for dual-modality whole heart segmentation.
Collapse
Affiliation(s)
- Bo Chen
- School of Mechanical, Electrical and Information Engineering, Shandong University, Weihai 264209, China; (B.C.); (Y.L.); (J.L.)
| | - Yongbo Li
- School of Mechanical, Electrical and Information Engineering, Shandong University, Weihai 264209, China; (B.C.); (Y.L.); (J.L.)
| | - Jiacheng Liu
- School of Mechanical, Electrical and Information Engineering, Shandong University, Weihai 264209, China; (B.C.); (Y.L.); (J.L.)
| | - Fei Yang
- School of Mechanical, Electrical and Information Engineering, Shandong University, Weihai 264209, China; (B.C.); (Y.L.); (J.L.)
| | - Lei Zhang
- Department of Diagnostic Radiology and Nuclear Medicine, University of Maryland School of Medicine, Baltimore, MD 21201, USA;
| |
Collapse
|
8
|
Yao T, Pajaziti E, Quail M, Schievano S, Steeden J, Muthurangu V. Image2Flow: A proof-of-concept hybrid image and graph convolutional neural network for rapid patient-specific pulmonary artery segmentation and CFD flow field calculation from 3D cardiac MRI data. PLoS Comput Biol 2024; 20:e1012231. [PMID: 38900817 PMCID: PMC11218942 DOI: 10.1371/journal.pcbi.1012231] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/22/2024] [Revised: 07/02/2024] [Accepted: 06/06/2024] [Indexed: 06/22/2024] Open
Abstract
Computational fluid dynamics (CFD) can be used for non-invasive evaluation of hemodynamics. However, its routine use is limited by labor-intensive manual segmentation, CFD mesh creation, and time-consuming simulation. This study aims to train a deep learning model to both generate patient-specific volume-meshes of the pulmonary artery from 3D cardiac MRI data and directly estimate CFD flow fields. This proof-of-concept study used 135 3D cardiac MRIs from both a public and private dataset. The pulmonary arteries in the MRIs were manually segmented and converted into volume-meshes. CFD simulations were performed on ground truth meshes and interpolated onto point-point correspondent meshes to create the ground truth dataset. The dataset was split 110/10/15 for training, validation, and testing. Image2Flow, a hybrid image and graph convolutional neural network, was trained to transform a pulmonary artery template to patient-specific anatomy and CFD values, taking a specific inlet velocity as an additional input. Image2Flow was evaluated in terms of segmentation, and the accuracy of predicted CFD was assessed using node-wise comparisons. In addition, the ability of Image2Flow to respond to increasing inlet velocities was also evaluated. Image2Flow achieved excellent segmentation accuracy with a median Dice score of 0.91 (IQR: 0.86-0.92). The median node-wise normalized absolute error for pressure and velocity magnitude was 11.75% (IQR: 9.60-15.30%) and 9.90% (IQR: 8.47-11.90), respectively. Image2Flow also showed an expected response to increased inlet velocities with increasing pressure and velocity values. This proof-of-concept study has shown that it is possible to simultaneously perform patient-specific volume-mesh based segmentation and pressure and flow field estimation using Image2Flow. Image2Flow completes segmentation and CFD in ~330ms, which is ~5000 times faster than manual methods, making it more feasible in a clinical environment.
Collapse
Affiliation(s)
- Tina Yao
- Institute of Cardiovascular Science, University College London, London, United Kingdom
| | - Endrit Pajaziti
- Institute of Cardiovascular Science, University College London, London, United Kingdom
| | - Michael Quail
- Institute of Cardiovascular Science, University College London, London, United Kingdom
| | - Silvia Schievano
- Institute of Cardiovascular Science, University College London, London, United Kingdom
| | - Jennifer Steeden
- Institute of Cardiovascular Science, University College London, London, United Kingdom
| | - Vivek Muthurangu
- Institute of Cardiovascular Science, University College London, London, United Kingdom
| |
Collapse
|
9
|
Qin Y, Qin X, Zhang J, Guo X. Artificial intelligence: The future for multimodality imaging of right ventricle. Int J Cardiol 2024; 404:131970. [PMID: 38490268 DOI: 10.1016/j.ijcard.2024.131970] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 11/26/2023] [Revised: 03/05/2024] [Accepted: 03/12/2024] [Indexed: 03/17/2024]
Abstract
The crucial pathophysiological and prognostic roles of the right ventricle in various diseases have been well-established. Nonetheless, conventional cardiovascular imaging modalities are frequently associated with intrinsic limitations when evaluating right ventricular (RV) morphology and function. The integration of artificial intelligence (AI) in multimodality imaging presents a promising avenue to circumvent these obstacles, paving the way for future fully automated imaging paradigms. This review aimed to address the current challenges faced by clinicians and researchers in integrating RV imaging and AI technology, to provide a comprehensive overview of the current applications of AI in RV imaging, and to offer insights into future directions, opportunities, and potential challenges in this rapidly advancing field.
Collapse
Affiliation(s)
- Yuhan Qin
- Department of Cardiology, Peking Union Medical College Hospital, Chinese Academy of Medical Science and Peking Union Medical College, Beijing, China
| | - Xiaohan Qin
- Department of Cardiology, Peking Union Medical College Hospital, Chinese Academy of Medical Science and Peking Union Medical College, Beijing, China
| | - Jing Zhang
- Department of Cardiology, Peking Union Medical College Hospital, Chinese Academy of Medical Science and Peking Union Medical College, Beijing, China
| | - Xiaoxiao Guo
- Department of Cardiology, Peking Union Medical College Hospital, Chinese Academy of Medical Science and Peking Union Medical College, Beijing, China.
| |
Collapse
|
10
|
Summerfield N, Morris E, Banerjee S, He Q, Ghanem AI, Zhu S, Zhao J, Dong M, Glide-Hurst C. Enhancing Precision in Cardiac Segmentation for Magnetic Resonance-Guided Radiation Therapy Through Deep Learning. Int J Radiat Oncol Biol Phys 2024:S0360-3016(24)00671-0. [PMID: 38797498 DOI: 10.1016/j.ijrobp.2024.05.013] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/03/2024] [Revised: 04/25/2024] [Accepted: 05/17/2024] [Indexed: 05/29/2024]
Abstract
PURPOSE Cardiac substructure dose metrics are more strongly linked to late cardiac morbidities than to whole-heart metrics. Magnetic resonance (MR)-guided radiation therapy (MRgRT) enables substructure visualization during daily localization, allowing potential for enhanced cardiac sparing. We extend a publicly available state-of-the-art deep learning framework, "No New" U-Net, to incorporate self-distillation (nnU-Net.wSD) for substructure segmentation for MRgRT. METHODS AND MATERIALS Eighteen (institute A) patients who underwent thoracic or abdominal radiation therapy on a 0.35 T MR-guided linear accelerator were retrospectively evaluated. On each image, 1 of 2 radiation oncologists delineated reference contours of 12 cardiac substructures (chambers, great vessels, and coronary arteries) used to train (n = 10), validate (n = 3), and test (n = 5) nnU-Net.wSD by leveraging a teacher-student network and comparing it to standard 3-dimensional U-Net. The impact of using simulation data or including 3 to 4 daily images for augmentation during training was evaluated for nnU-Net.wSD. Geometric metrics (Dice similarity coefficient, mean distance to agreement, and 95% Hausdorff distance), visual inspection, and clinical dose-volume histograms were evaluated. To determine generalizability, institute A's model was tested on an unlabeled data set from institute B (n = 22) and evaluated via consensus scoring and volume comparisons. RESULTS nnU-Net.wSD yielded a Dice similarity coefficient (reported mean ± SD) of 0.65 ± 0.25 across the 12 substructures (chambers, 0.85 ± 0.05; great vessels, 0.67 ± 0.19; and coronary arteries, 0.33 ± 0.16; mean distance to agreement, <3 mm; mean 95% Hausdorff distance, <9 mm) while outperforming the 3-dimensional U-Net (0.583 ± 0.28; P <.01). Leveraging fractionated data for augmentation improved over a single MR simulation time point (0.579 ± 0.29; P <.01). Predicted contours yielded dose-volume histograms that closely matched those of the clinical treatment plans where mean and maximum (ie, dose to 0.03 cc) doses deviated by 0.32 ± 0.5 Gy and 1.42 ± 2.6 Gy, respectively. There were no statistically significant differences between institute A and B volumes (P >.05) for 11 of 12 substructures, with larger volumes requiring minor changes and coronary arteries exhibiting more variability. CONCLUSIONS This work is a critical step toward rapid and reliable cardiac substructure segmentation to improve cardiac sparing in low-field MRgRT.
Collapse
Affiliation(s)
- Nicholas Summerfield
- Department of Medical Physics, University of Wisconsin-Madison, Madison, Wisconsin; Department of Human Oncology, University of Wisconsin-Madison, Madison, Wisconsin
| | - Eric Morris
- Department of Radiation Oncology, Washington University of Medicine in St. Louis, St. Louis, Missouri
| | - Soumyanil Banerjee
- Department of Computer Science, Wayne State University, Detroit, Michigan
| | - Qisheng He
- Department of Computer Science, Wayne State University, Detroit, Michigan
| | - Ahmed I Ghanem
- Department of Radiation Oncology, Henry Ford Cancer Institute, Detroit, Michigan; Alexandria Department of Clinical Oncology, Faculty of Medicine, Alexandria University, Alexandria, Egypt
| | - Simeng Zhu
- Department of Radiation Oncology, The Ohio State University, Columbus, Ohio
| | - Jiwei Zhao
- Department of Biostatistics and Medical Informatics, University of Wisconsin-Madison, Madison, Wisconsin
| | - Ming Dong
- Department of Computer Science, Wayne State University, Detroit, Michigan
| | - Carri Glide-Hurst
- Department of Medical Physics, University of Wisconsin-Madison, Madison, Wisconsin; Department of Human Oncology, University of Wisconsin-Madison, Madison, Wisconsin.
| |
Collapse
|
11
|
Zhi Y, Bie H, Wang J, Ren L. Masked autoencoders with generalizable self-distillation for skin lesion segmentation. Med Biol Eng Comput 2024:10.1007/s11517-024-03086-z. [PMID: 38653880 DOI: 10.1007/s11517-024-03086-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/04/2023] [Accepted: 03/29/2024] [Indexed: 04/25/2024]
Abstract
In the field of skin lesion image segmentation, accurate identification and partitioning of diseased regions is of vital importance for in-depth analysis of skin cancer. Self-supervised learning, i.e., MAE, has emerged as a potent force in the medical imaging domain, which autonomously learns and extracts latent features from unlabeled data, thereby yielding pre-trained models that greatly assist downstream tasks. To encourage pre-trained models to more comprehensively learn the global structural and local detail information inherent in dermoscopy images, we introduce a Teacher-Student architecture, named TEDMAE, by incorporating a self-distillation mechanism, it learns holistic image feature information to improve the generalizable global knowledge learning of the student MAE model. To make the image features learned by the model suitable for unknown test images, two optimization strategies are, Exterior Conversion Augmentation (EC) utilizes random convolutional kernels and linear interpolation to effectively transform the input image into one with the same shape but altered intensities and textures, while Dynamic Feature Generation (DF) employs a nonlinear attention mechanism for feature merging, enhancing the expressive power of the features, are proposed to enhance the generalizability of global features learned by the teacher model, thereby improving the overall generalization capability of the pre-trained models. Experimental results from the three public skin disease datasets, ISIC2019, ISIC2017, and PH2 indicate that our proposed TEDMAE method outperforms several similar approaches. Specifically, TEDMAE demonstrated optimal segmentation and generalization performance on the ISIC2017 and PH2 datasets, with Dice scores reaching 82.1% and 91.2%, respectively. The best Jaccard values were 72.6% and 84.5%, while the optimal HD95% values were 13.0% and 8.9%, respectively.
Collapse
Affiliation(s)
- Yichen Zhi
- Department of Intelligent Media Computing Center, School of Artificial Intelligence, Beijing University of Posts and Telecommunications, Beijing, 100876, People's Republic of China
| | - Hongxia Bie
- Department of Intelligent Media Computing Center, School of Artificial Intelligence, Beijing University of Posts and Telecommunications, Beijing, 100876, People's Republic of China.
| | - Jiali Wang
- Department of Intelligent Media Computing Center, School of Artificial Intelligence, Beijing University of Posts and Telecommunications, Beijing, 100876, People's Republic of China
| | - Lihan Ren
- Department of Intelligent Media Computing Center, School of Artificial Intelligence, Beijing University of Posts and Telecommunications, Beijing, 100876, People's Republic of China
| |
Collapse
|
12
|
Vernikouskaya I, Bertsche D, Metze P, Schneider LM, Rasche V. Multi-network approach for image segmentation in non-contrast enhanced cardiac 3D MRI of arrhythmic patients. Comput Med Imaging Graph 2024; 113:102340. [PMID: 38277768 DOI: 10.1016/j.compmedimag.2024.102340] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/06/2023] [Revised: 01/16/2024] [Accepted: 01/16/2024] [Indexed: 01/28/2024]
Abstract
Left atrial appendage (LAA) is the source of thrombi formation in more than 90% of strokes in patients with nonvalvular atrial fibrillation. Catheter-based LAA occlusion is being increasingly applied as a treatment strategy to prevent stroke. Anatomical complexity of LAA makes percutaneous occlusion commonly performed under transesophageal echocardiography (TEE) and X-ray (XR) guidance especially challenging. Image fusion techniques integrating 3D anatomical models derived from pre-procedural imaging into the live XR fluoroscopy can be applied to guide each step of the LAA closure. Cardiac magnetic resonance (CMR) imaging gains in importance for radiation-free evaluation of cardiac morphology as alternative to gold-standard TEE or computed tomography angiography (CTA). Manual delineation of cardiac structures from non-contrast enhanced CMR is, however, labor-intensive, tedious, and challenging due to the rather low contrast. Additionally, arrhythmia often impairs the image quality in ECG synchronized acquisitions causing blurring and motion artifacts. Thus, for cardiac segmentation in arrhythmic patients, there is a strong need for an automated image segmentation method. Deep learning-based methods have shown great promise in medical image analysis achieving superior performance in various imaging modalities and different clinical applications. Fully-convolutional neural networks (CNNs), especially U-Net, have become the method of choice for cardiac segmentation. In this paper, we propose an approach for automatic segmentation of cardiac structures from non-contrast enhanced CMR images of arrhythmic patients based on CNNs implemented in a multi-stage pipeline. Two-stage implementation allows subdividing the task into localization of the relevant cardiac structures and segmentation of these structures from the cropped sub-regions obtained from previous step leading to efficient and effective way of automated cardiac segmentation.
Collapse
Affiliation(s)
- Ina Vernikouskaya
- Department of Internal Medicine II, Ulm University Medical Center, Ulm, Germany.
| | - Dagmar Bertsche
- Department of Internal Medicine II, Ulm University Medical Center, Ulm, Germany.
| | - Patrick Metze
- Department of Internal Medicine II, Ulm University Medical Center, Ulm, Germany.
| | - Leonhard M Schneider
- Department of Internal Medicine II, Ulm University Medical Center, Ulm, Germany.
| | - Volker Rasche
- Department of Internal Medicine II, Ulm University Medical Center, Ulm, Germany.
| |
Collapse
|
13
|
Huang Y, Yang J, Sun Q, Yuan Y, Li H, Hou Y. Multi-residual 2D network integrating spatial correlation for whole heart segmentation. Comput Biol Med 2024; 172:108261. [PMID: 38508056 DOI: 10.1016/j.compbiomed.2024.108261] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/19/2023] [Revised: 02/21/2024] [Accepted: 03/06/2024] [Indexed: 03/22/2024]
Abstract
Whole heart segmentation (WHS) has significant clinical value for cardiac anatomy, modeling, and analysis of cardiac function. This study aims to address the WHS accuracy on cardiac CT images, as well as the fast inference speed and low graphics processing unit (GPU) memory consumption required by practical clinical applications. Thus, we propose a multi-residual two-dimensional (2D) network integrating spatial correlation for WHS. The network performs slice-by-slice segmentation on three-dimensional cardiac CT images in a 2D encoder-decoder manner. In the network, a convolutional long short-term memory skip connection module is designed to perform spatial correlation feature extraction on the feature maps at different resolutions extracted by the sub-modules of the pre-trained ResNet-based encoder. Moreover, a decoder based on the multi-residual module is designed to analyze the extracted features from the perspectives of multi-scale and channel attention, thereby accurately delineating the various substructures of the heart. The proposed method is verified on a dataset of the multi-modality WHS challenge, an in-house WHS dataset, and a dataset of the abdominal organ segmentation challenge. The dice, Jaccard, average symmetric surface distance, Hausdorff distance, inference time, and maximum GPU memory of the WHS are 0.914, 0.843, 1.066 mm, 15.778 mm, 9.535 s, and 1905 MB, respectively. The proposed network has high accuracy, fast inference speed, minimal GPU memory consumption, strong robustness, and good generalization. It can be deployed to clinical practical applications for WHS and can be effectively extended and applied to other multi-organ segmentation fields. The source code is publicly available at https://github.com/nancy1984yan/MultiResNet-SC.
Collapse
Affiliation(s)
- Yan Huang
- Key Laboratory of Intelligent Computing in Medical Image, Ministry of Education, Northeastern University, Shenyang, Liaoning, China; School of Computer Science and Engineering, Northeastern University, Shenyang, Liaoning, China
| | - Jinzhu Yang
- Key Laboratory of Intelligent Computing in Medical Image, Ministry of Education, Northeastern University, Shenyang, Liaoning, China; School of Computer Science and Engineering, Northeastern University, Shenyang, Liaoning, China; National Frontiers Science Center for Industrial Intelligence and Systems Optimization, Shenyang, Liaoning, China.
| | - Qi Sun
- Key Laboratory of Intelligent Computing in Medical Image, Ministry of Education, Northeastern University, Shenyang, Liaoning, China; School of Computer Science and Engineering, Northeastern University, Shenyang, Liaoning, China
| | - Yuliang Yuan
- Key Laboratory of Intelligent Computing in Medical Image, Ministry of Education, Northeastern University, Shenyang, Liaoning, China; School of Computer Science and Engineering, Northeastern University, Shenyang, Liaoning, China
| | - Honghe Li
- Key Laboratory of Intelligent Computing in Medical Image, Ministry of Education, Northeastern University, Shenyang, Liaoning, China; School of Computer Science and Engineering, Northeastern University, Shenyang, Liaoning, China
| | - Yang Hou
- Department of Radiology, Shengjing Hospital of China Medical University, Shenyang, Liaoning, China
| |
Collapse
|
14
|
Bhattaru A, Rojulpote C, Vidula M, Duda J, Maclean MT, Swago S, Thompson E, Gee J, Pieretti J, Drachman B, Cohen A, Dorbala S, Bravo PE, Witschey WR. Deep learning approach for automated segmentation of myocardium using bone scintigraphy single-photon emission computed tomography/computed tomography in patients with suspected cardiac amyloidosis. J Nucl Cardiol 2024; 33:101809. [PMID: 38307160 DOI: 10.1016/j.nuclcard.2024.101809] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/01/2023] [Revised: 08/02/2023] [Accepted: 08/02/2023] [Indexed: 02/04/2024]
Abstract
BACKGROUND We employed deep learning to automatically detect myocardial bone-seeking uptake as a marker of transthyretin cardiac amyloid cardiomyopathy (ATTR-CM) in patients undergoing 99mTc-pyrophosphate (PYP) or hydroxydiphosphonate (HDP) single-photon emission computed tomography (SPECT)/computed tomography (CT). METHODS We identified a primary cohort of 77 subjects at Brigham and Women's Hospital and a validation cohort of 93 consecutive patients imaged at the University of Pennsylvania who underwent SPECT/CT with PYP and HDP, respectively, for evaluation of ATTR-CM. Global heart regions of interest (ROIs) were traced on CT axial slices from the apex of the ventricle to the carina. Myocardial images were visually scored as grade 0 (no uptake), 1 (uptakeribs). A 2D U-net architecture was used to develop whole-heart segmentations for CT scans. Uptake was determined by calculating a heart-to-blood pool (HBP) ratio between the maximal counts value of the total heart region and the maximal counts value of the most superior ROI. RESULTS Deep learning and ground truth segmentations were comparable (p=0.63). A total of 42 (55%) patients had abnormal myocardial uptake on visual assessment. Automated quantification of the mean HBP ratio in the primary cohort was 3.1±1.4 versus 1.4±0.2 (p<0.01) for patients with positive and negative cardiac uptake, respectively. The model had 100% accuracy in the primary cohort and 98% in the validation cohort. CONCLUSION We have developed a highly accurate diagnostic tool for automatically segmenting and identifying myocardial uptake suggestive of ATTR-CM.
Collapse
Affiliation(s)
- Abhijit Bhattaru
- Department of Radiology, University of Pennsylvania, Philadelphia, PA, USA; Department of Cardiology, University of Pennsylvania, Philadelphia, PA, USA; Department of Medicine, Rutgers New Jersey Medical School, Newark, NJ, USA
| | - Chaitanya Rojulpote
- Department of Radiology, University of Pennsylvania, Philadelphia, PA, USA; Department of Cardiology, University of Pennsylvania, Philadelphia, PA, USA
| | - Mahesh Vidula
- Department of Cardiology, University of Pennsylvania, Philadelphia, PA, USA
| | - Jeffrey Duda
- Department of Radiology, University of Pennsylvania, Philadelphia, PA, USA
| | - Matthew T Maclean
- Department of Radiology, University of Pennsylvania, Philadelphia, PA, USA
| | - Sophia Swago
- Department of Radiology, University of Pennsylvania, Philadelphia, PA, USA
| | - Elizabeth Thompson
- Department of Radiology, University of Pennsylvania, Philadelphia, PA, USA
| | - James Gee
- Department of Radiology, University of Pennsylvania, Philadelphia, PA, USA
| | - Janice Pieretti
- Department of Cardiology, University of Pennsylvania, Philadelphia, PA, USA
| | - Brian Drachman
- Department of Cardiology, University of Pennsylvania, Philadelphia, PA, USA
| | - Adam Cohen
- Department of Oncology, University of Pennsylvania, Philadelphia, PA, USA
| | - Sharmila Dorbala
- Department of Radiology, Brigham and Women's Hospital, Boston, MA, USA
| | - Paco E Bravo
- Department of Radiology, University of Pennsylvania, Philadelphia, PA, USA; Department of Cardiology, University of Pennsylvania, Philadelphia, PA, USA.
| | - Walter R Witschey
- Department of Radiology, University of Pennsylvania, Philadelphia, PA, USA
| |
Collapse
|
15
|
Kumari S, Singh P. Deep learning for unsupervised domain adaptation in medical imaging: Recent advancements and future perspectives. Comput Biol Med 2024; 170:107912. [PMID: 38219643 DOI: 10.1016/j.compbiomed.2023.107912] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/20/2023] [Revised: 11/02/2023] [Accepted: 12/24/2023] [Indexed: 01/16/2024]
Abstract
Deep learning has demonstrated remarkable performance across various tasks in medical imaging. However, these approaches primarily focus on supervised learning, assuming that the training and testing data are drawn from the same distribution. Unfortunately, this assumption may not always hold true in practice. To address these issues, unsupervised domain adaptation (UDA) techniques have been developed to transfer knowledge from a labeled domain to a related but unlabeled domain. In recent years, significant advancements have been made in UDA, resulting in a wide range of methodologies, including feature alignment, image translation, self-supervision, and disentangled representation methods, among others. In this paper, we provide a comprehensive literature review of recent deep UDA approaches in medical imaging from a technical perspective. Specifically, we categorize current UDA research in medical imaging into six groups and further divide them into finer subcategories based on the different tasks they perform. We also discuss the respective datasets used in the studies to assess the divergence between the different domains. Finally, we discuss emerging areas and provide insights and discussions on future research directions to conclude this survey.
Collapse
Affiliation(s)
- Suruchi Kumari
- Department of Computer Science and Engineering, Indian Institute of Technology Roorkee, India.
| | - Pravendra Singh
- Department of Computer Science and Engineering, Indian Institute of Technology Roorkee, India.
| |
Collapse
|
16
|
Zhang M, Ye Z, Yuan E, Lv X, Zhang Y, Tan Y, Xia C, Tang J, Huang J, Li Z. Imaging-based deep learning in kidney diseases: recent progress and future prospects. Insights Imaging 2024; 15:50. [PMID: 38360904 PMCID: PMC10869329 DOI: 10.1186/s13244-024-01636-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/19/2023] [Accepted: 01/27/2024] [Indexed: 02/17/2024] Open
Abstract
Kidney diseases result from various causes, which can generally be divided into neoplastic and non-neoplastic diseases. Deep learning based on medical imaging is an established methodology for further data mining and an evolving field of expertise, which provides the possibility for precise management of kidney diseases. Recently, imaging-based deep learning has been widely applied to many clinical scenarios of kidney diseases including organ segmentation, lesion detection, differential diagnosis, surgical planning, and prognosis prediction, which can provide support for disease diagnosis and management. In this review, we will introduce the basic methodology of imaging-based deep learning and its recent clinical applications in neoplastic and non-neoplastic kidney diseases. Additionally, we further discuss its current challenges and future prospects and conclude that achieving data balance, addressing heterogeneity, and managing data size remain challenges for imaging-based deep learning. Meanwhile, the interpretability of algorithms, ethical risks, and barriers of bias assessment are also issues that require consideration in future development. We hope to provide urologists, nephrologists, and radiologists with clear ideas about imaging-based deep learning and reveal its great potential in clinical practice.Critical relevance statement The wide clinical applications of imaging-based deep learning in kidney diseases can help doctors to diagnose, treat, and manage patients with neoplastic or non-neoplastic renal diseases.Key points• Imaging-based deep learning is widely applied to neoplastic and non-neoplastic renal diseases.• Imaging-based deep learning improves the accuracy of the delineation, diagnosis, and evaluation of kidney diseases.• The small dataset, various lesion sizes, and so on are still challenges for deep learning.
Collapse
Affiliation(s)
- Meng Zhang
- Department of Radiology, West China Hospital, Sichuan University, No. 37 Guoxue Alley, Chengdu, 610041, China
- Medical Equipment Innovation Research Center, West China Hospital, Sichuan University, No. 37 Guoxue Alley, Chengdu, 610041, China
- Med+X Center for Manufacturing, West China Hospital, Sichuan University, No. 37 Guoxue Alley, Chengdu, 610041, China
| | - Zheng Ye
- Department of Radiology, West China Hospital, Sichuan University, No. 37 Guoxue Alley, Chengdu, 610041, China
| | - Enyu Yuan
- Department of Radiology, West China Hospital, Sichuan University, No. 37 Guoxue Alley, Chengdu, 610041, China
| | - Xinyang Lv
- Department of Radiology, West China Hospital, Sichuan University, No. 37 Guoxue Alley, Chengdu, 610041, China
| | - Yiteng Zhang
- Department of Radiology, West China Hospital, Sichuan University, No. 37 Guoxue Alley, Chengdu, 610041, China
| | - Yuqi Tan
- Department of Radiology, West China Hospital, Sichuan University, No. 37 Guoxue Alley, Chengdu, 610041, China
| | - Chunchao Xia
- Department of Radiology, West China Hospital, Sichuan University, No. 37 Guoxue Alley, Chengdu, 610041, China
| | - Jing Tang
- Department of Radiology, West China Hospital, Sichuan University, No. 37 Guoxue Alley, Chengdu, 610041, China.
| | - Jin Huang
- Medical Equipment Innovation Research Center, West China Hospital, Sichuan University, No. 37 Guoxue Alley, Chengdu, 610041, China.
- Med+X Center for Manufacturing, West China Hospital, Sichuan University, No. 37 Guoxue Alley, Chengdu, 610041, China.
| | - Zhenlin Li
- Department of Radiology, West China Hospital, Sichuan University, No. 37 Guoxue Alley, Chengdu, 610041, China.
| |
Collapse
|
17
|
Brown AL, Sexton ZA, Hu Z, Yang W, Marsden AL. Computational approaches for mechanobiology in cardiovascular development and diseases. Curr Top Dev Biol 2024; 156:19-50. [PMID: 38556423 DOI: 10.1016/bs.ctdb.2024.01.006] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 04/02/2024]
Abstract
The cardiovascular development in vertebrates evolves in response to genetic and mechanical cues. The dynamic interplay among mechanics, cell biology, and anatomy continually shapes the hydraulic networks, characterized by complex, non-linear changes in anatomical structure and blood flow dynamics. To better understand this interplay, a diverse set of molecular and computational tools has been used to comprehensively study cardiovascular mechanobiology. With the continual advancement of computational capacity and numerical techniques, cardiovascular simulation is increasingly vital in both basic science research for understanding developmental mechanisms and disease etiologies, as well as in clinical studies aimed at enhancing treatment outcomes. This review provides an overview of computational cardiovascular modeling. Beginning with the fundamental concepts of computational cardiovascular modeling, it navigates through the applications of computational modeling in investigating mechanobiology during cardiac development. Second, the article illustrates the utility of computational hemodynamic modeling in the context of treatment planning for congenital heart diseases. It then delves into the predictive potential of computational models for elucidating tissue growth and remodeling processes. In closing, we outline prevailing challenges and future prospects, underscoring the transformative impact of computational cardiovascular modeling in reshaping cardiovascular science and clinical practice.
Collapse
Affiliation(s)
- Aaron L Brown
- Department of Mechanical Engineering, Stanford University, Stanford, CA, United States
| | - Zachary A Sexton
- Department of Bioengineering, Stanford University, Stanford, CA, United States
| | - Zinan Hu
- Department of Mechanical Engineering, Stanford University, Stanford, CA, United States
| | - Weiguang Yang
- Department of Pediatrics, Stanford University, Stanford, CA, United States
| | - Alison L Marsden
- Department of Bioengineering, Stanford University, Stanford, CA, United States; Department of Pediatrics, Stanford University, Stanford, CA, United States.
| |
Collapse
|
18
|
He Y, Ge R, Qi X, Chen Y, Wu J, Coatrieux JL, Yang G, Li S. Learning Better Registration to Learn Better Few-Shot Medical Image Segmentation: Authenticity, Diversity, and Robustness. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 2024; 35:2588-2601. [PMID: 35895657 DOI: 10.1109/tnnls.2022.3190452] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/15/2023]
Abstract
In this work, we address the task of few-shot medical image segmentation (MIS) with a novel proposed framework based on the learning registration to learn segmentation (LRLS) paradigm. To cope with the limitations of lack of authenticity, diversity, and robustness in the existing LRLS frameworks, we propose the better registration better segmentation (BRBS) framework with three main contributions that are experimentally shown to have substantial practical merit. First, we improve the authenticity in the registration-based generation program and propose the knowledge consistency constraint strategy that constrains the registration network to learn according to the domain knowledge. It brings the semantic-aligned and topology-preserved registration, thus allowing the generation program to output new data with great space and style authenticity. Second, we deeply studied the diversity of the generation process and propose the space-style sampling program, which introduces the modeling of the transformation path of style and space change between few atlases and numerous unlabeled images into the generation program. Therefore, the sampling on the transformation paths provides much more diverse space and style features to the generated data effectively improving the diversity. Third, we first highlight the robustness in the learning of segmentation in the LRLS paradigm and propose the mix misalignment regularization, which simulates the misalignment distortion and constrains the network to reduce the fitting degree of misaligned regions. Therefore, it builds regularization for these regions improving the robustness of segmentation learning. Without any bells and whistles, our approach achieves a new state-of-the-art performance in few-shot MIS on two challenging tasks that outperform the existing LRLS-based few-shot methods. We believe that this novel and effective framework will provide a powerful few-shot benchmark for the field of medical image and efficiently reduce the costs of medical image research. All of our code will be made publicly available online.
Collapse
|
19
|
Milosevic M, Jin Q, Singh A, Amal S. Applications of AI in multi-modal imaging for cardiovascular disease. FRONTIERS IN RADIOLOGY 2024; 3:1294068. [PMID: 38283302 PMCID: PMC10811170 DOI: 10.3389/fradi.2023.1294068] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 09/14/2023] [Accepted: 12/22/2023] [Indexed: 01/30/2024]
Abstract
Data for healthcare is diverse and includes many different modalities. Traditional approaches to Artificial Intelligence for cardiovascular disease were typically limited to single modalities. With the proliferation of diverse datasets and new methods in AI, we are now able to integrate different modalities, such as magnetic resonance scans, computerized tomography scans, echocardiography, x-rays, and electronic health records. In this paper, we review research from the last 5 years in applications of AI to multi-modal imaging. There have been many promising results in registration, segmentation, and fusion of different magnetic resonance imaging modalities with each other and computer tomography scans, but there are still many challenges that need to be addressed. Only a few papers have addressed modalities such as x-ray, echocardiography, or non-imaging modalities. As for prediction or classification tasks, there have only been a couple of papers that use multiple modalities in the cardiovascular domain. Furthermore, no models have been implemented or tested in real world cardiovascular clinical settings.
Collapse
Affiliation(s)
- Marko Milosevic
- Roux Institute, Northeastern University, Portland, ME, United States
| | - Qingchu Jin
- Roux Institute, Northeastern University, Portland, ME, United States
| | - Akarsh Singh
- College of Engineering, Northeastern University, Boston, MA, United States
| | - Saeed Amal
- Roux Institute, Northeastern University, Portland, ME, United States
| |
Collapse
|
20
|
Muffoletto M, Xu H, Kunze KP, Neji R, Botnar R, Prieto C, Rückert D, Young AA. Combining generative modelling and semi-supervised domain adaptation for whole heart cardiovascular magnetic resonance angiography segmentation. J Cardiovasc Magn Reson 2023; 25:80. [PMID: 38124106 PMCID: PMC10734115 DOI: 10.1186/s12968-023-00981-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/14/2023] [Accepted: 11/12/2023] [Indexed: 12/23/2023] Open
Abstract
BACKGROUND Quantification of three-dimensional (3D) cardiac anatomy is important for the evaluation of cardiovascular diseases. Changes in anatomy are indicative of remodeling processes as the heart tissue adapts to disease. Although robust segmentation methods exist for computed tomography angiography (CTA), few methods exist for whole-heart cardiovascular magnetic resonance angiograms (CMRA) which are more challenging due to variable contrast, lower signal to noise ratio and a limited amount of labeled data. METHODS Two state-of-the-art unsupervised generative deep learning domain adaptation architectures, generative adversarial networks and variational auto-encoders, were applied to 3D whole heart segmentation of both conventional (n = 20) and high-resolution (n = 45) CMRA (target) images, given segmented CTA (source) images for training. An additional supervised loss function was implemented to improve performance given 10%, 20% and 30% segmented CMRA cases. A fully supervised nn-UNet trained on the given CMRA segmentations was used as the benchmark. RESULTS The addition of a small number of segmented CMRA training cases substantially improved performance in both generative architectures in both standard and high-resolution datasets. Compared with the nn-UNet benchmark, the generative methods showed substantially better performance in the case of limited labelled cases. On the standard CMRA dataset, an average 12% (adversarial method) and 10% (variational method) improvement in Dice score was obtained. CONCLUSIONS Unsupervised domain-adaptation methods for CMRA segmentation can be boosted by the addition of a small number of supervised target training cases. When only few labelled cases are available, semi-supervised generative modelling is superior to supervised methods.
Collapse
Affiliation(s)
- Marica Muffoletto
- School of Biomedical Engineering and Imaging Sciences, King's College, St Thomas' Hospital, 4th Floor Lambeth Wing, Westminster Bridge, London, SW1 7EH, UK.
| | - Hao Xu
- School of Biomedical Engineering and Imaging Sciences, King's College, St Thomas' Hospital, 4th Floor Lambeth Wing, Westminster Bridge, London, SW1 7EH, UK
| | - Karl P Kunze
- School of Biomedical Engineering and Imaging Sciences, King's College, St Thomas' Hospital, 4th Floor Lambeth Wing, Westminster Bridge, London, SW1 7EH, UK
- MR Research Collaborations, Siemens Healthcare Limited, Frimley, UK
| | - Radhouene Neji
- School of Biomedical Engineering and Imaging Sciences, King's College, St Thomas' Hospital, 4th Floor Lambeth Wing, Westminster Bridge, London, SW1 7EH, UK
- MR Research Collaborations, Siemens Healthcare Limited, Frimley, UK
| | - René Botnar
- School of Biomedical Engineering and Imaging Sciences, King's College, St Thomas' Hospital, 4th Floor Lambeth Wing, Westminster Bridge, London, SW1 7EH, UK
| | - Claudia Prieto
- School of Biomedical Engineering and Imaging Sciences, King's College, St Thomas' Hospital, 4th Floor Lambeth Wing, Westminster Bridge, London, SW1 7EH, UK
| | - Daniel Rückert
- Biomedical Image Analysis Group, Department of Computing, Imperial College London, London, UK
- Institute for Artificial Intelligence and Informatics in Medicine, Klinikum Rechts der Isar, Technical University of Munich, Munich, Germany
| | - Alistair A Young
- School of Biomedical Engineering and Imaging Sciences, King's College, St Thomas' Hospital, 4th Floor Lambeth Wing, Westminster Bridge, London, SW1 7EH, UK
| |
Collapse
|
21
|
Xu X, Jia Q, Yuan H, Qiu H, Dong Y, Xie W, Yao Z, Zhang J, Nie Z, Li X, Shi Y, Zou JY, Huang M, Zhuang J. A clinically applicable AI system for diagnosis of congenital heart diseases based on computed tomography images. Med Image Anal 2023; 90:102953. [PMID: 37734140 DOI: 10.1016/j.media.2023.102953] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/19/2022] [Revised: 08/22/2023] [Accepted: 09/01/2023] [Indexed: 09/23/2023]
Abstract
Congenital heart disease (CHD) is the most common type of birth defect. Without timely detection and treatment, approximately one-third of children with CHD would die in the infant period. However, due to the complicated heart structures, early diagnosis of CHD and its types is quite challenging, even for experienced radiologists. Here, we present an artificial intelligence (AI) system that achieves a comparable performance of human experts in the critical task of classifying 17 categories of CHD types. We collected the first-large CT dataset from three different CT machines, including more than 3750 CHD patients over 14 years. Experimental results demonstrate that it can achieve diagnosis accuracy (86.03%) comparable with junior cardiovascular radiologists (86.27%) in a World Health Organization appointed research and cooperation center in China on most types of CHD, and obtains a higher sensitivity (82.91%) than junior cardiovascular radiologists (76.18%). The accuracy of the combination of our AI system (97.20%) and senior radiologists achieves comparable results to that of junior radiologists and senior radiologists (97.16%) which is the current clinical routine. Our AI system can further provide 3D visualization of hearts to senior radiologists for interpretation and flexible review, surgeons for precise intuition of heart structures, and clinicians for more precise outcome prediction. We demonstrate the potential of our model to be integrated into current clinic practice to improve the diagnosis of CHD globally, especially in regions where experienced radiologists can be scarce.
Collapse
Affiliation(s)
- Xiaowei Xu
- Guangdong Provincial Key Laboratory of South China Structural Heart Disease, Guangdong Provincial People's Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, 510080, China; Guangdong Cardiovascular Institute, Guangdong Provincial People's Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, 510080, China
| | - Qianjun Jia
- Guangdong Provincial Key Laboratory of South China Structural Heart Disease, Guangdong Provincial People's Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, 510080, China; Guangdong Cardiovascular Institute, Guangdong Provincial People's Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, 510080, China; Department of Catheterization Lab, Guangdong Provincial People's Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, China
| | - Haiyun Yuan
- Guangdong Provincial Key Laboratory of South China Structural Heart Disease, Guangdong Provincial People's Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, 510080, China; Guangdong Cardiovascular Institute, Guangdong Provincial People's Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, 510080, China; Department of Cardiovascular Surgery, Guangdong Provincial People's Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, 510080, China
| | - Hailong Qiu
- Guangdong Provincial Key Laboratory of South China Structural Heart Disease, Guangdong Provincial People's Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, 510080, China; Guangdong Cardiovascular Institute, Guangdong Provincial People's Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, 510080, China; Department of Cardiovascular Surgery, Guangdong Provincial People's Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, 510080, China
| | - Yuhao Dong
- Guangdong Provincial Key Laboratory of South China Structural Heart Disease, Guangdong Provincial People's Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, 510080, China; Guangdong Cardiovascular Institute, Guangdong Provincial People's Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, 510080, China; Department of Catheterization Lab, Guangdong Provincial People's Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, China
| | - Wen Xie
- Guangdong Provincial Key Laboratory of South China Structural Heart Disease, Guangdong Provincial People's Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, 510080, China; Guangdong Cardiovascular Institute, Guangdong Provincial People's Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, 510080, China; Department of Cardiovascular Surgery, Guangdong Provincial People's Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, 510080, China
| | - Zeyang Yao
- Guangdong Provincial Key Laboratory of South China Structural Heart Disease, Guangdong Provincial People's Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, 510080, China; Guangdong Cardiovascular Institute, Guangdong Provincial People's Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, 510080, China; Department of Cardiovascular Surgery, Guangdong Provincial People's Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, 510080, China
| | - Jiawei Zhang
- Guangdong Provincial Key Laboratory of South China Structural Heart Disease, Guangdong Provincial People's Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, 510080, China; Guangdong Cardiovascular Institute, Guangdong Provincial People's Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, 510080, China
| | - Zhiqaing Nie
- Guangdong Cardiovascular Institute, Guangdong Provincial People's Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, 510080, China
| | - Xiaomeng Li
- Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology, Hong Kong Special Administrative Region
| | - Yiyu Shi
- Computer Science and Engineering, University of Notre Dame, IN, 46656, USA
| | - James Y Zou
- Department of Computer Science, Stanford University, Stanford, CA, 94305, USA; Department of Electrical Engineering, Stanford University, Stanford, CA, 94305, USA.
| | - Meiping Huang
- Guangdong Provincial Key Laboratory of South China Structural Heart Disease, Guangdong Provincial People's Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, 510080, China; Guangdong Cardiovascular Institute, Guangdong Provincial People's Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, 510080, China; Department of Catheterization Lab, Guangdong Provincial People's Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, China.
| | - Jian Zhuang
- Guangdong Provincial Key Laboratory of South China Structural Heart Disease, Guangdong Provincial People's Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, 510080, China; Guangdong Cardiovascular Institute, Guangdong Provincial People's Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, 510080, China; Department of Cardiovascular Surgery, Guangdong Provincial People's Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, 510080, China.
| |
Collapse
|
22
|
Zhang M, Wu Y, Zhang H, Qin Y, Zheng H, Tang W, Arnold C, Pei C, Yu P, Nan Y, Yang G, Walsh S, Marshall DC, Komorowski M, Wang P, Guo D, Jin D, Wu Y, Zhao S, Chang R, Zhang B, Lu X, Qayyum A, Mazher M, Su Q, Wu Y, Liu Y, Zhu Y, Yang J, Pakzad A, Rangelov B, Estepar RSJ, Espinosa CC, Sun J, Yang GZ, Gu Y. Multi-site, Multi-domain Airway Tree Modeling. Med Image Anal 2023; 90:102957. [PMID: 37716199 DOI: 10.1016/j.media.2023.102957] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/27/2023] [Revised: 06/07/2023] [Accepted: 09/04/2023] [Indexed: 09/18/2023]
Abstract
Open international challenges are becoming the de facto standard for assessing computer vision and image analysis algorithms. In recent years, new methods have extended the reach of pulmonary airway segmentation that is closer to the limit of image resolution. Since EXACT'09 pulmonary airway segmentation, limited effort has been directed to the quantitative comparison of newly emerged algorithms driven by the maturity of deep learning based approaches and extensive clinical efforts for resolving finer details of distal airways for early intervention of pulmonary diseases. Thus far, public annotated datasets are extremely limited, hindering the development of data-driven methods and detailed performance evaluation of new algorithms. To provide a benchmark for the medical imaging community, we organized the Multi-site, Multi-domain Airway Tree Modeling (ATM'22), which was held as an official challenge event during the MICCAI 2022 conference. ATM'22 provides large-scale CT scans with detailed pulmonary airway annotation, including 500 CT scans (300 for training, 50 for validation, and 150 for testing). The dataset was collected from different sites and it further included a portion of noisy COVID-19 CTs with ground-glass opacity and consolidation. Twenty-three teams participated in the entire phase of the challenge and the algorithms for the top ten teams are reviewed in this paper. Both quantitative and qualitative results revealed that deep learning models embedded with the topological continuity enhancement achieved superior performance in general. ATM'22 challenge holds as an open-call design, the training data and the gold standard evaluation are available upon successful registration via its homepage (https://atm22.grand-challenge.org/).
Collapse
Affiliation(s)
- Minghui Zhang
- Institute of Medical Robotics, Shanghai Jiao Tong University, Shanghai, 200240, China; Institute of Image Processing and Pattern Recognition, Shanghai Jiao Tong University, Shanghai, 200240, China; Department of Automation, Shanghai Jiao Tong University, Shanghai, 200240, China
| | - Yangqian Wu
- Institute of Medical Robotics, Shanghai Jiao Tong University, Shanghai, 200240, China; Institute of Image Processing and Pattern Recognition, Shanghai Jiao Tong University, Shanghai, 200240, China; Department of Automation, Shanghai Jiao Tong University, Shanghai, 200240, China
| | - Hanxiao Zhang
- Institute of Medical Robotics, Shanghai Jiao Tong University, Shanghai, 200240, China
| | - Yulei Qin
- Institute of Medical Robotics, Shanghai Jiao Tong University, Shanghai, 200240, China
| | - Hao Zheng
- Institute of Medical Robotics, Shanghai Jiao Tong University, Shanghai, 200240, China
| | - Wen Tang
- InferVision Medical Technology Co., Ltd., Beijing, China
| | | | - Chenhao Pei
- InferVision Medical Technology Co., Ltd., Beijing, China
| | - Pengxin Yu
- InferVision Medical Technology Co., Ltd., Beijing, China
| | - Yang Nan
- Imperial College London, London, UK
| | | | | | | | | | - Puyang Wang
- Alibaba DAMO Academy, 969 West Wen Yi Road, Hangzhou, Zhejiang, China
| | - Dazhou Guo
- Alibaba DAMO Academy USA, 860 Washington Street, 8F, NY, USA
| | - Dakai Jin
- Alibaba DAMO Academy USA, 860 Washington Street, 8F, NY, USA
| | - Ya'nan Wu
- College of Medicine and Biological Information Engineering, Northeastern University, Shenyang, China
| | - Shuiqing Zhao
- College of Medicine and Biological Information Engineering, Northeastern University, Shenyang, China
| | - Runsheng Chang
- College of Medicine and Biological Information Engineering, Northeastern University, Shenyang, China
| | - Boyu Zhang
- A.I R&D Center, Sanmed Biotech Inc., No. 266 Tongchang Road, Xiangzhou District, Zhuhai, Guangdong, China
| | - Xing Lu
- A.I R&D Center, Sanmed Biotech Inc., T220 Trade st. SanDiego, CA, USA
| | - Abdul Qayyum
- ENIB, UMR CNRS 6285 LabSTICC, Brest, 29238, France
| | - Moona Mazher
- Department of Computer Engineering and Mathematics, University Rovira I Virgili, Tarragona, Spain
| | - Qi Su
- Shanghai Jiao Tong University, Shanghai, China
| | - Yonghuang Wu
- School of Information Science and Technology, Fudan University, Shanghai, China
| | - Ying'ao Liu
- University of Science and Technology of China, Hefei, Anhui, China
| | | | - Jiancheng Yang
- Dianei Technology, Shanghai, China; EPFL, Lausanne, Switzerland
| | - Ashkan Pakzad
- Medical Physics and Biomedical Engineering Department, University College London, London, UK
| | - Bojidar Rangelov
- Center for Medical Image Computing, University College London, London, UK
| | | | | | - Jiayuan Sun
- Department of Respiratory and Critical Care Medicine, Department of Respiratory Endoscopy, Shanghai Chest Hospital, Shanghai, China.
| | - Guang-Zhong Yang
- Institute of Medical Robotics, Shanghai Jiao Tong University, Shanghai, 200240, China.
| | - Yun Gu
- Institute of Medical Robotics, Shanghai Jiao Tong University, Shanghai, 200240, China; Institute of Image Processing and Pattern Recognition, Shanghai Jiao Tong University, Shanghai, 200240, China; Department of Automation, Shanghai Jiao Tong University, Shanghai, 200240, China.
| |
Collapse
|
23
|
Feng R, Deb B, Ganesan P, Tjong FVY, Rogers AJ, Ruipérez-Campillo S, Somani S, Clopton P, Baykaner T, Rodrigo M, Zou J, Haddad F, Zahari M, Narayan SM. Segmenting computed tomograms for cardiac ablation using machine learning leveraged by domain knowledge encoding. Front Cardiovasc Med 2023; 10:1189293. [PMID: 37849936 PMCID: PMC10577270 DOI: 10.3389/fcvm.2023.1189293] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/18/2023] [Accepted: 09/18/2023] [Indexed: 10/19/2023] Open
Abstract
Background Segmentation of computed tomography (CT) is important for many clinical procedures including personalized cardiac ablation for the management of cardiac arrhythmias. While segmentation can be automated by machine learning (ML), it is limited by the need for large, labeled training data that may be difficult to obtain. We set out to combine ML of cardiac CT with domain knowledge, which reduces the need for large training datasets by encoding cardiac geometry, which we then tested in independent datasets and in a prospective study of atrial fibrillation (AF) ablation. Methods We mathematically represented atrial anatomy with simple geometric shapes and derived a model to parse cardiac structures in a small set of N = 6 digital hearts. The model, termed "virtual dissection," was used to train ML to segment cardiac CT in N = 20 patients, then tested in independent datasets and in a prospective study. Results In independent test cohorts (N = 160) from 2 Institutions with different CT scanners, atrial structures were accurately segmented with Dice scores of 96.7% in internal (IQR: 95.3%-97.7%) and 93.5% in external (IQR: 91.9%-94.7%) test data, with good agreement with experts (r = 0.99; p < 0.0001). In a prospective study of 42 patients at ablation, this approach reduced segmentation time by 85% (2.3 ± 0.8 vs. 15.0 ± 6.9 min, p < 0.0001), yet provided similar Dice scores to experts (93.9% (IQR: 93.0%-94.6%) vs. 94.4% (IQR: 92.8%-95.7%), p = NS). Conclusions Encoding cardiac geometry using mathematical models greatly accelerated training of ML to segment CT, reducing the need for large training sets while retaining accuracy in independent test data. Combining ML with domain knowledge may have broad applications.
Collapse
Affiliation(s)
- Ruibin Feng
- Department of Medicine and Cardiovascular Institute, Stanford University, Stanford, CA, United States
| | - Brototo Deb
- Department of Medicine and Cardiovascular Institute, Stanford University, Stanford, CA, United States
| | - Prasanth Ganesan
- Department of Medicine and Cardiovascular Institute, Stanford University, Stanford, CA, United States
| | - Fleur V. Y. Tjong
- Department of Medicine and Cardiovascular Institute, Stanford University, Stanford, CA, United States
- Heart Center, Department of Clinical and Experimental Cardiology, Amsterdam UMC, University of Amsterdam, Amsterdam, Netherlands
| | - Albert J. Rogers
- Department of Medicine and Cardiovascular Institute, Stanford University, Stanford, CA, United States
| | - Samuel Ruipérez-Campillo
- Department of Medicine and Cardiovascular Institute, Stanford University, Stanford, CA, United States
- Bioengineering Department, University of California, Berkeley, Berkeley, CA, United States
| | - Sulaiman Somani
- Department of Medicine and Cardiovascular Institute, Stanford University, Stanford, CA, United States
| | - Paul Clopton
- Department of Medicine and Cardiovascular Institute, Stanford University, Stanford, CA, United States
| | - Tina Baykaner
- Department of Medicine and Cardiovascular Institute, Stanford University, Stanford, CA, United States
| | - Miguel Rodrigo
- Department of Medicine and Cardiovascular Institute, Stanford University, Stanford, CA, United States
- CoMMLab, Universitat Politècnica de València, Valencia, Spain
| | - James Zou
- Department of Biomedical Data Science, Stanford University, Stanford, CA, United States
| | - Francois Haddad
- Department of Medicine and Cardiovascular Institute, Stanford University, Stanford, CA, United States
| | - Matei Zahari
- Department of Computer Science, Stanford University, Stanford, CA, United States
| | - Sanjiv M. Narayan
- Department of Medicine and Cardiovascular Institute, Stanford University, Stanford, CA, United States
| |
Collapse
|
24
|
Li D, Peng Y, Sun J, Guo Y. Unsupervised deep consistency learning adaptation network for cardiac cross-modality structural segmentation. Med Biol Eng Comput 2023; 61:2713-2732. [PMID: 37450212 DOI: 10.1007/s11517-023-02833-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/24/2022] [Accepted: 04/05/2023] [Indexed: 07/18/2023]
Abstract
Deep neural networks have recently been succeessful in the field of medical image segmentation; however, they are typically subject to performance degradation problems when well-trained models are tested in another new domain with different data distributions. Given that annotated cross-domain images may inaccessible, unsupervised domain adaptation methods that transfer learnable information from annotated source domains to unannotated target domains with different distributions have attracted substantial attention. Many methods leverage image-level or pixel-level translation networks to align domain-invariant information and mitigate domain shift issues. However, These methods rarely perform well when there is a large domain gap. A new unsupervised deep consistency learning adaptation network, which adopts input space consistency learning and output space consistency learning to realize unsupervised domain adaptation and cardiac structural segmentation, is introduced in this paper The framework mainly includes a domain translation path and a cross-modality segmentation path. In domain translation path, a symmetric alignment generator network with attention to cross-modality features and anatomy is introduced to align bidirectional domain features. In the segmentation path, entropy map minimization, output probability map minimization and segmentation prediction minimization are leveraged to align the output space features. The model conducts supervised learning to extract source domain features and conducts unsupervised deep consistency learning to extract target domain features. Through experimental testing on two challenging cross-modality segmentation tasks, our method has robust performance compared to that of previous methods. Furthermore, ablation experiments are conducted to confirm the effectiveness of our framework.
Collapse
Affiliation(s)
- Dapeng Li
- College of Computer Science and Engineering, Shandong University of Science and Technology, Qingdao, 266590, Shandong, China
| | - Yanjun Peng
- College of Computer Science and Engineering, Shandong University of Science and Technology, Qingdao, 266590, Shandong, China.
- Shandong Province Key Laboratory of Wisdom Mining Information Technology, Shandong University of Science and Technology, Qingdao, 266590, Shandong, China.
| | - Jindong Sun
- College of Computer Science and Engineering, Shandong University of Science and Technology, Qingdao, 266590, Shandong, China
| | - Yanfei Guo
- College of Computer Science and Engineering, Shandong University of Science and Technology, Qingdao, 266590, Shandong, China
| |
Collapse
|
25
|
Li X, Luo G, Wang W, Wang K, Li S. Curriculum label distribution learning for imbalanced medical image segmentation. Med Image Anal 2023; 89:102911. [PMID: 37542795 DOI: 10.1016/j.media.2023.102911] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/28/2022] [Revised: 04/27/2023] [Accepted: 07/25/2023] [Indexed: 08/07/2023]
Abstract
Label distribution learning (LDL) has the potential to resolve boundary ambiguity in semantic segmentation tasks. However, existing LDL-based segmentation methods suffer from severe label distribution imbalance: the ambiguous label distributions contain a small fraction of the data, while the unambiguous label distributions occupy the majority of the data. The imbalanced label distributions induce model-biased distribution learning and make it challenging to accurately predict ambiguous pixels. In this paper, we propose a curriculum label distribution learning (CLDL) framework to address the above data imbalance problem by performing a novel task-oriented curriculum learning strategy. Firstly, the region label distribution learning (R-LDL) is proposed to construct more balanced label distributions and improves the imbalanced model learning. Secondly, a novel learning curriculum (TCL) is proposed to enable easy-to-hard learning in LDL-based segmentation by decomposing the segmentation task into multiple label distribution estimation tasks. Thirdly, the prior perceiving module (PPM) is proposed to effectively connect easy and hard learning stages based on the priors generated from easier stages. Benefiting from the balanced label distribution construction and prior perception, the proposed CLDL effectively conducts a curriculum learning-based LDL and significantly improves the imbalanced learning. We evaluated the proposed CLDL using the publicly available BRATS2018 and MM-WHS2017 datasets. The experimental results demonstrate that our method significantly improves different segmentation metrics compared to many state-of-the-art methods. The code will be available.1.
Collapse
Affiliation(s)
- Xiangyu Li
- School of Computer Science and Technology, Harbin Institute of Technology, Harbin, China
| | - Gongning Luo
- School of Computer Science and Technology, Harbin Institute of Technology, Harbin, China.
| | - Wei Wang
- School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen, China.
| | - Kuanquan Wang
- School of Computer Science and Technology, Harbin Institute of Technology, Harbin, China
| | - Shuo Li
- Department of Biomedical Engineering, Case Western Reserve University, Cleveland, OH 44106, USA.
| |
Collapse
|
26
|
Wang R, Zhou Q, Zheng G. EDRL: Entropy-guided disentangled representation learning for unsupervised domain adaptation in semantic segmentation. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2023; 240:107729. [PMID: 37531690 DOI: 10.1016/j.cmpb.2023.107729] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/09/2022] [Revised: 07/15/2023] [Accepted: 07/19/2023] [Indexed: 08/04/2023]
Abstract
BACKGROUND AND OBJECTIVE Deep learning-based approaches are excellent at learning from large amounts of data, but can be poor at generalizing the learned knowledge to testing datasets with domain shift, i.e., when there exists distribution discrepancy between the training dataset (source domain) and the testing dataset (target domain). In this paper, we investigate unsupervised domain adaptation (UDA) techniques to train a cross-domain segmentation method which is robust to domain shift, eliminating the requirement of any annotations on the target domain. METHODS To this end, we propose an Entropy-guided Disentangled Representation Learning, referred as EDRL, for UDA in semantic segmentation. Concretely, we synergistically integrate image alignment via disentangled representation learning with feature alignment via entropy-based adversarial learning into one network, which is trained end-to-end. We additionally introduce a dynamic feature selection mechanism via soft gating, which helps to further enhance the task-specific feature alignment. We validate the proposed method on two publicly available datasets: the CT-MR dataset and the multi-sequence cardiac MR (MS-CMR) dataset. RESULTS On both datasets, our method achieved better results than the state-of-the-art (SOTA) methods. Specifically, on the CT-MR dataset, our method achieved an average DSC of 84.8% when taking CT as the source domain and MR as the target domain, and an average DSC of 84.0% when taking MR as the source domain and CT as the target domain. CONCLUSIONS Results from comprehensive experiments demonstrate the efficacy of the proposed EDRL model for cross-domain medical image segmentation.
Collapse
Affiliation(s)
- Runze Wang
- Institute of Medical Robotics, School of Biomedical Engineering, Shanghai Jiao Tong University, No. 800, Dongchuan Road, Shanghai, 200240, China
| | - Qin Zhou
- Institute of Medical Robotics, School of Biomedical Engineering, Shanghai Jiao Tong University, No. 800, Dongchuan Road, Shanghai, 200240, China
| | - Guoyan Zheng
- Institute of Medical Robotics, School of Biomedical Engineering, Shanghai Jiao Tong University, No. 800, Dongchuan Road, Shanghai, 200240, China.
| |
Collapse
|
27
|
Wan K, Li L, Jia D, Gao S, Qian W, Wu Y, Lin H, Mu X, Gao X, Wang S, Wu F, Zhuang X. Multi-target landmark detection with incomplete images via reinforcement learning and shape prior embedding. Med Image Anal 2023; 89:102875. [PMID: 37441881 DOI: 10.1016/j.media.2023.102875] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/10/2023] [Revised: 05/05/2023] [Accepted: 06/13/2023] [Indexed: 07/15/2023]
Abstract
Medical images are generally acquired with limited field-of-view (FOV), which could lead to incomplete regions of interest (ROI), and thus impose a great challenge on medical image analysis. This is particularly evident for the learning-based multi-target landmark detection, where algorithms could be misleading to learn primarily the variation of background due to the varying FOV, failing the detection of targets. Based on learning a navigation policy, instead of predicting targets directly, reinforcement learning (RL)-based methods have the potential to tackle this challenge in an efficient manner. Inspired by this, in this work we propose a multi-agent RL framework for simultaneous multi-target landmark detection. This framework is aimed to learn from incomplete or (and) complete images to form an implicit knowledge of global structure, which is consolidated during the training stage for the detection of targets from either complete or incomplete test images. To further explicitly exploit the global structural information from incomplete images, we propose to embed a shape model into the RL process. With this prior knowledge, the proposed RL model can not only localize dozens of targets simultaneously, but also work effectively and robustly in the presence of incomplete images. We validated the applicability and efficacy of the proposed method on various multi-target detection tasks with incomplete images from practical clinics, using body dual-energy X-ray absorptiometry (DXA), cardiac MRI and head CT datasets. Results showed that our method could predict whole set of landmarks with incomplete training images up to 80% missing proportion (average distance error 2.29 cm on body DXA), and could detect unseen landmarks in regions with missing image information outside FOV of target images (average distance error 6.84 mm on 3D half-head CT). Our code will be released via https://zmiclab.github.io/projects.html.
Collapse
Affiliation(s)
- Kaiwen Wan
- School of Data Science, Fudan University, Shanghai, 200433, China
| | - Lei Li
- Institute of Biomedical Engineering, University of Oxford, Oxford, UK
| | - Dengqiang Jia
- School of Naval Architecture, Ocean and Civil Engineering, Shanghai Jiao Tong University, Shanghai, China; Hong Kong Centre for Cerebro-Cardiovascular Health Engineering (COCHE), Hong Kong, China
| | - Shangqi Gao
- School of Data Science, Fudan University, Shanghai, 200433, China
| | - Wei Qian
- Shanghai Institute of Nutrition and Health, University of Chinese Academy of Sciences, Chinese Academy of Sciences, Shanghai 200031, China
| | - Yingzhi Wu
- Department of Plastic Surgery, Huashan Hospital, Fudan University, Shanghai 200040, China
| | - Huandong Lin
- Department of Endocrinology and Metabolism, Zhong Shan Hospital, Fudan University, 200032 Shanghai, China
| | - Xiongzheng Mu
- Department of Plastic Surgery, Huashan Hospital, Fudan University, Shanghai 200040, China
| | - Xin Gao
- Department of Endocrinology and Metabolism, Zhong Shan Hospital, Fudan University, 200032 Shanghai, China
| | - Sijia Wang
- Shanghai Institute of Nutrition and Health, University of Chinese Academy of Sciences, Chinese Academy of Sciences, Shanghai 200031, China
| | - Fuping Wu
- Nuffield Department of Population Health, University of Oxford, Oxford, UK
| | - Xiahai Zhuang
- School of Data Science, Fudan University, Shanghai, 200433, China.
| |
Collapse
|
28
|
Kazemimoghadam M, Yang Z, Chen M, Ma L, Lu W, Gu X. Leveraging global binary masks for structure segmentation in medical images. Phys Med Biol 2023; 68:10.1088/1361-6560/acf2e2. [PMID: 37607564 PMCID: PMC10511220 DOI: 10.1088/1361-6560/acf2e2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/12/2022] [Accepted: 08/22/2023] [Indexed: 08/24/2023]
Abstract
Deep learning (DL) models for medical image segmentation are highly influenced by intensity variations of input images and lack generalization due to primarily utilizing pixels' intensity information for inference. Acquiring sufficient training data is another challenge limiting models' applications. Here, we proposed to leverage the consistency of organs' anatomical position and shape information in medical images. We introduced a framework leveraging recurring anatomical patterns through global binary masks for organ segmentation. Two scenarios were studied: (1) global binary masks were the only input for the U-Net based model, forcing exclusively encoding organs' position and shape information for rough segmentation or localization. (2) Global binary masks were incorporated as an additional channel providing position/shape clues to mitigate training data scarcity. Two datasets of the brain and heart computed tomography (CT) images with their ground-truth were split into (26:10:10) and (12:3:5) for training, validation, and test respectively. The two scenarios were evaluated using full training split as well as reduced subsets of training data. In scenario (1), training exclusively on global binary masks led to Dice scores of 0.77 ± 0.06 and 0.85 ± 0.04 for the brain and heart structures respectively. Average Euclidian distance of 3.12 ± 1.43 mm and 2.5 ± 0.93 mm were obtained relative to the center of mass of the ground truth for the brain and heart structures respectively. The outcomes indicated encoding a surprising degree of position and shape information through global binary masks. In scenario (2), incorporating global binary masks led to significantly higher accuracy relative to the model trained on only CT images in small subsets of training data; the performance improved by 4.3%-125.3% and 1.3%-48.1% for 1-8 training cases of the brain and heart datasets respectively. The findings imply the advantages of utilizing global binary masks for building models that are robust to image intensity variations as well as an effective approach to boost performance when access to labeled training data is highly limited.
Collapse
Affiliation(s)
- Mahdieh Kazemimoghadam
- Department of Radiation Oncology, the University of Texas Southwestern Medical Center, Dallas TX, 75390 USA
| | - Zi Yang
- Department of Radiation Oncology, the University of Texas Southwestern Medical Center, Dallas TX, 75390 USA
| | - Mingli Chen
- Department of Radiation Oncology, the University of Texas Southwestern Medical Center, Dallas TX, 75390 USA
| | - Lin Ma
- Department of Radiation Oncology, the University of Texas Southwestern Medical Center, Dallas TX, 75390 USA
| | - Weiguo Lu
- Department of Radiation Oncology, the University of Texas Southwestern Medical Center, Dallas TX, 75390 USA
| | - Xuejun Gu
- Department of Radiation Oncology, the University of Texas Southwestern Medical Center, Dallas TX, 75390 USA
- Department of Radiation Oncology, Stanford University, Stanford, CA 94305
| |
Collapse
|
29
|
Liu H, Zhuang Y, Song E, Xu X, Ma G, Cetinkaya C, Hung CC. A modality-collaborative convolution and transformer hybrid network for unpaired multi-modal medical image segmentation with limited annotations. Med Phys 2023; 50:5460-5478. [PMID: 36864700 DOI: 10.1002/mp.16338] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/28/2022] [Revised: 02/07/2023] [Accepted: 02/22/2023] [Indexed: 03/04/2023] Open
Abstract
BACKGROUND Multi-modal learning is widely adopted to learn the latent complementary information between different modalities in multi-modal medical image segmentation tasks. Nevertheless, the traditional multi-modal learning methods require spatially well-aligned and paired multi-modal images for supervised training, which cannot leverage unpaired multi-modal images with spatial misalignment and modality discrepancy. For training accurate multi-modal segmentation networks using easily accessible and low-cost unpaired multi-modal images in clinical practice, unpaired multi-modal learning has received comprehensive attention recently. PURPOSE Existing unpaired multi-modal learning methods usually focus on the intensity distribution gap but ignore the scale variation problem between different modalities. Besides, within existing methods, shared convolutional kernels are frequently employed to capture common patterns in all modalities, but they are typically inefficient at learning global contextual information. On the other hand, existing methods highly rely on a large number of labeled unpaired multi-modal scans for training, which ignores the practical scenario when labeled data is limited. To solve the above problems, we propose a modality-collaborative convolution and transformer hybrid network (MCTHNet) using semi-supervised learning for unpaired multi-modal segmentation with limited annotations, which not only collaboratively learns modality-specific and modality-invariant representations, but also could automatically leverage extensive unlabeled scans for improving performance. METHODS We make three main contributions to the proposed method. First, to alleviate the intensity distribution gap and scale variation problems across modalities, we develop a modality-specific scale-aware convolution (MSSC) module that can adaptively adjust the receptive field sizes and feature normalization parameters according to the input. Secondly, we propose a modality-invariant vision transformer (MIViT) module as the shared bottleneck layer for all modalities, which implicitly incorporates convolution-like local operations with the global processing of transformers for learning generalizable modality-invariant representations. Third, we design a multi-modal cross pseudo supervision (MCPS) method for semi-supervised learning, which enforces the consistency between the pseudo segmentation maps generated by two perturbed networks to acquire abundant annotation information from unlabeled unpaired multi-modal scans. RESULTS Extensive experiments are performed on two unpaired CT and MR segmentation datasets, including a cardiac substructure dataset derived from the MMWHS-2017 dataset and an abdominal multi-organ dataset consisting of the BTCV and CHAOS datasets. Experiment results show that our proposed method significantly outperforms other existing state-of-the-art methods under various labeling ratios, and achieves a comparable segmentation performance close to single-modal methods with fully labeled data by only leveraging a small portion of labeled data. Specifically, when the labeling ratio is 25%, our proposed method achieves overall mean DSC values of 78.56% and 76.18% in cardiac and abdominal segmentation, respectively, which significantly improves the average DSC value of two tasks by 12.84% compared to single-modal U-Net models. CONCLUSIONS Our proposed method is beneficial for reducing the annotation burden of unpaired multi-modal medical images in clinical applications.
Collapse
Affiliation(s)
- Hong Liu
- School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan, China
| | - Yuzhou Zhuang
- School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan, China
| | - Enmin Song
- School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan, China
| | - Xiangyang Xu
- School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan, China
| | - Guangzhi Ma
- School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan, China
| | - Coskun Cetinkaya
- Center for Machine Vision and Security Research, Kennesaw State University, Kennesaw, Georgia, USA
| | - Chih-Cheng Hung
- Center for Machine Vision and Security Research, Kennesaw State University, Kennesaw, Georgia, USA
| |
Collapse
|
30
|
Li L, Ding W, Huang L, Zhuang X, Grau V. Multi-modality cardiac image computing: A survey. Med Image Anal 2023; 88:102869. [PMID: 37384950 DOI: 10.1016/j.media.2023.102869] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/25/2022] [Revised: 05/01/2023] [Accepted: 06/12/2023] [Indexed: 07/01/2023]
Abstract
Multi-modality cardiac imaging plays a key role in the management of patients with cardiovascular diseases. It allows a combination of complementary anatomical, morphological and functional information, increases diagnosis accuracy, and improves the efficacy of cardiovascular interventions and clinical outcomes. Fully-automated processing and quantitative analysis of multi-modality cardiac images could have a direct impact on clinical research and evidence-based patient management. However, these require overcoming significant challenges including inter-modality misalignment and finding optimal methods to integrate information from different modalities. This paper aims to provide a comprehensive review of multi-modality imaging in cardiology, the computing methods, the validation strategies, the related clinical workflows and future perspectives. For the computing methodologies, we have a favored focus on the three tasks, i.e., registration, fusion and segmentation, which generally involve multi-modality imaging data, either combining information from different modalities or transferring information across modalities. The review highlights that multi-modality cardiac imaging data has the potential of wide applicability in the clinic, such as trans-aortic valve implantation guidance, myocardial viability assessment, and catheter ablation therapy and its patient selection. Nevertheless, many challenges remain unsolved, such as missing modality, modality selection, combination of imaging and non-imaging data, and uniform analysis and representation of different modalities. There is also work to do in defining how the well-developed techniques fit in clinical workflows and how much additional and relevant information they introduce. These problems are likely to continue to be an active field of research and the questions to be answered in the future.
Collapse
Affiliation(s)
- Lei Li
- Department of Engineering Science, University of Oxford, Oxford, UK.
| | - Wangbin Ding
- College of Physics and Information Engineering, Fuzhou University, Fuzhou, China
| | - Liqin Huang
- College of Physics and Information Engineering, Fuzhou University, Fuzhou, China
| | - Xiahai Zhuang
- School of Data Science, Fudan University, Shanghai, China
| | - Vicente Grau
- Department of Engineering Science, University of Oxford, Oxford, UK
| |
Collapse
|
31
|
Inomata S, Yoshimura T, Tang M, Ichikawa S, Sugimori H. Estimation of Left and Right Ventricular Ejection Fractions from cine-MRI Using 3D-CNN. SENSORS (BASEL, SWITZERLAND) 2023; 23:6580. [PMID: 37514888 PMCID: PMC10384911 DOI: 10.3390/s23146580] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/23/2023] [Revised: 07/17/2023] [Accepted: 07/20/2023] [Indexed: 07/30/2023]
Abstract
Cardiac function indices must be calculated using tracing from short-axis images in cine-MRI. A 3D-CNN (convolutional neural network) that adds time series information to images can estimate cardiac function indices without tracing using images with known values and cardiac cycles as the input. Since the short-axis image depicts the left and right ventricles, it is unclear which motion feature is captured. This study aims to estimate the indices by learning the short-axis images and the known left and right ventricular ejection fractions and to confirm the accuracy and whether each index is captured as a feature. A total of 100 patients with publicly available short-axis cine images were used. The dataset was divided into training:test = 8:2, and a regression model was built by training with the 3D-ResNet50. Accuracy was assessed using a five-fold cross-validation. The correlation coefficient, MAE (mean absolute error), and RMSE (root mean squared error) were determined as indices of accuracy evaluation. The mean correlation coefficient of the left ventricular ejection fraction was 0.80, MAE was 9.41, and RMSE was 12.26. The mean correlation coefficient of the right ventricular ejection fraction was 0.56, MAE was 11.35, and RMSE was 14.95. The correlation coefficient was considerably higher for the left ventricular ejection fraction. Regression modeling using the 3D-CNN indicated that the left ventricular ejection fraction was estimated more accurately, and left ventricular systolic function was captured as a feature.
Collapse
Affiliation(s)
- Soichiro Inomata
- Graduate School of Health Sciences, Hokkaido University, Sapporo 060-0812, Japan
| | - Takaaki Yoshimura
- Department of Health Sciences and Technology, Faculty of Health Sciences, Hokkaido University, Sapporo 060-0812, Japan
- Department of Medical Physics, Hokkaido University Hospital, Sapporo 060-8648, Japan
- Global Center for Biomedical Science and Engineering, Faculty of Medicine, Hokkaido University, Sapporo 060-8648, Japan
- Clinical AI Human Resources Development Program, Faculty of Medicine, Hokkaido University, Sapporo 060-8648, Japan
| | - Minghui Tang
- Clinical AI Human Resources Development Program, Faculty of Medicine, Hokkaido University, Sapporo 060-8648, Japan
- Department of Diagnostic Imaging, Faculty of Medicine and Graduate School of Medicine, Hokkaido University, Sapporo 060-8638, Japan
| | - Shota Ichikawa
- Department of Radiological Technology, School of Health Sciences, Faculty of Medicine, Niigata University, Niigata 951-8518, Japan
- Institute for Research Administration, Niigata University, Niigata 950-2181, Japan
| | - Hiroyuki Sugimori
- Global Center for Biomedical Science and Engineering, Faculty of Medicine, Hokkaido University, Sapporo 060-8648, Japan
- Clinical AI Human Resources Development Program, Faculty of Medicine, Hokkaido University, Sapporo 060-8648, Japan
- Department of Biomedical Science and Engineering, Faculty of Health Sciences, Hokkaido University, Sapporo 060-0812, Japan
| |
Collapse
|
32
|
Liu P, Zheng G. CVCL: Context-aware Voxel-wise Contrastive Learning for label-efficient multi-organ segmentation. Comput Biol Med 2023; 160:106995. [PMID: 37187134 DOI: 10.1016/j.compbiomed.2023.106995] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/26/2022] [Revised: 04/02/2023] [Accepted: 05/01/2023] [Indexed: 05/17/2023]
Abstract
Despite the significant performance improvement on multi-organ segmentation with supervised deep learning-based methods, the label-hungry nature hinders their applications in practical disease diagnosis and treatment planning. Due to the challenges in obtaining expert-level accurate, densely annotated multi-organ datasets, label-efficient segmentation, such as partially supervised segmentation trained on partially labeled datasets or semi-supervised medical image segmentation, has attracted increasing attention recently. However, most of these methods suffer from the limitation that they neglect or underestimate the challenging unlabeled regions during model training. To this end, we propose a novel Context-aware Voxel-wise Contrastive Learning method, referred as CVCL, to take full advantage of both labeled and unlabeled information in label-scarce datasets for a performance improvement on multi-organ segmentation. Experimental results demonstrate that our proposed method achieves superior performance than other state-of-the-art methods.
Collapse
Affiliation(s)
- Peng Liu
- Institute of Medical Robotics, School of Biomedical Engineering, Shanghai Jiao Tong University, No. 800, Dongchuan Road, Shanghai, 200240, China
| | - Guoyan Zheng
- Institute of Medical Robotics, School of Biomedical Engineering, Shanghai Jiao Tong University, No. 800, Dongchuan Road, Shanghai, 200240, China.
| |
Collapse
|
33
|
Jafari M, Shoeibi A, Khodatars M, Ghassemi N, Moridian P, Alizadehsani R, Khosravi A, Ling SH, Delfan N, Zhang YD, Wang SH, Gorriz JM, Alinejad-Rokny H, Acharya UR. Automated diagnosis of cardiovascular diseases from cardiac magnetic resonance imaging using deep learning models: A review. Comput Biol Med 2023; 160:106998. [PMID: 37182422 DOI: 10.1016/j.compbiomed.2023.106998] [Citation(s) in RCA: 6] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/24/2022] [Revised: 03/01/2023] [Accepted: 04/28/2023] [Indexed: 05/16/2023]
Abstract
In recent years, cardiovascular diseases (CVDs) have become one of the leading causes of mortality globally. At early stages, CVDs appear with minor symptoms and progressively get worse. The majority of people experience symptoms such as exhaustion, shortness of breath, ankle swelling, fluid retention, and other symptoms when starting CVD. Coronary artery disease (CAD), arrhythmia, cardiomyopathy, congenital heart defect (CHD), mitral regurgitation, and angina are the most common CVDs. Clinical methods such as blood tests, electrocardiography (ECG) signals, and medical imaging are the most effective methods used for the detection of CVDs. Among the diagnostic methods, cardiac magnetic resonance imaging (CMRI) is increasingly used to diagnose, monitor the disease, plan treatment and predict CVDs. Coupled with all the advantages of CMR data, CVDs diagnosis is challenging for physicians as each scan has many slices of data, and the contrast of it might be low. To address these issues, deep learning (DL) techniques have been employed in the diagnosis of CVDs using CMR data, and much research is currently being conducted in this field. This review provides an overview of the studies performed in CVDs detection using CMR images and DL techniques. The introduction section examined CVDs types, diagnostic methods, and the most important medical imaging techniques. The following presents research to detect CVDs using CMR images and the most significant DL methods. Another section discussed the challenges in diagnosing CVDs from CMRI data. Next, the discussion section discusses the results of this review, and future work in CVDs diagnosis from CMR images and DL techniques are outlined. Finally, the most important findings of this study are presented in the conclusion section.
Collapse
Affiliation(s)
- Mahboobeh Jafari
- Internship in BioMedical Machine Learning Lab, The Graduate School of Biomedical Engineering, UNSW Sydney, Sydney, NSW, 2052, Australia
| | - Afshin Shoeibi
- Internship in BioMedical Machine Learning Lab, The Graduate School of Biomedical Engineering, UNSW Sydney, Sydney, NSW, 2052, Australia; Data Science and Computational Intelligence Institute, University of Granada, Spain.
| | - Marjane Khodatars
- Data Science and Computational Intelligence Institute, University of Granada, Spain
| | - Navid Ghassemi
- Internship in BioMedical Machine Learning Lab, The Graduate School of Biomedical Engineering, UNSW Sydney, Sydney, NSW, 2052, Australia
| | - Parisa Moridian
- Data Science and Computational Intelligence Institute, University of Granada, Spain
| | - Roohallah Alizadehsani
- Institute for Intelligent Systems Research and Innovation, Deakin University, Geelong, Australia
| | - Abbas Khosravi
- Institute for Intelligent Systems Research and Innovation, Deakin University, Geelong, Australia
| | - Sai Ho Ling
- Faculty of Engineering and IT, University of Technology Sydney (UTS), Australia
| | - Niloufar Delfan
- Faculty of Computer Engineering, Dept. of Artificial Intelligence Engineering, K. N. Toosi University of Technology, Tehran, Iran
| | - Yu-Dong Zhang
- School of Computing and Mathematical Sciences, University of Leicester, Leicester, UK
| | - Shui-Hua Wang
- School of Computing and Mathematical Sciences, University of Leicester, Leicester, UK
| | - Juan M Gorriz
- Data Science and Computational Intelligence Institute, University of Granada, Spain; Department of Psychiatry, University of Cambridge, UK
| | - Hamid Alinejad-Rokny
- BioMedical Machine Learning Lab, The Graduate School of Biomedical Engineering, UNSW Sydney, Sydney, NSW, 2052, Australia; UNSW Data Science Hub, The University of New South Wales, Sydney, NSW, 2052, Australia; Health Data Analytics Program, Centre for Applied Artificial Intelligence, Macquarie University, Sydney, 2109, Australia
| | - U Rajendra Acharya
- School of Mathematics, Physics and Computing, University of Southern Queensland, Springfield, Australia; Dept. of Biomedical Informatics and Medical Engineering, Asia University, Taichung, Taiwan
| |
Collapse
|
34
|
Wu F, Zhuang X. Minimizing Estimated Risks on Unlabeled Data: A New Formulation for Semi-Supervised Medical Image Segmentation. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2023; 45:6021-6036. [PMID: 36251907 DOI: 10.1109/tpami.2022.3215186] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/06/2023]
Abstract
Supervised segmentation can be costly, particularly in applications of biomedical image analysis where large scale manual annotations from experts are generally too expensive to be available. Semi-supervised segmentation, able to learn from both the labeled and unlabeled images, could be an efficient and effective alternative for such scenarios. In this work, we propose a new formulation based on risk minimization, which makes full use of the unlabeled images. Different from most of the existing approaches which solely explicitly guarantee the minimization of prediction risks from the labeled training images, the new formulation also considers the risks on unlabeled images. Particularly, this is achieved via an unbiased estimator, based on which we develop a general framework for semi-supervised image segmentation. We validate this framework on three medical image segmentation tasks, namely cardiac segmentation on ACDC2017, optic cup and disc segmentation on REFUGE dataset and 3D whole heart segmentation on MM-WHS dataset. Results show that the proposed estimator is effective, and the segmentation method achieves superior performance and demonstrates great potential compared to the other state-of-the-art approaches. Our code and data will be released via https://zmiclab.github.io/projects.html, once the manuscript is accepted for publication.
Collapse
|
35
|
Al Khalil Y, Amirrajab S, Lorenz C, Weese J, Pluim J, Breeuwer M. Reducing segmentation failures in cardiac MRI via late feature fusion and GAN-based augmentation. Comput Biol Med 2023; 161:106973. [PMID: 37209615 DOI: 10.1016/j.compbiomed.2023.106973] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/16/2022] [Revised: 04/05/2023] [Accepted: 04/22/2023] [Indexed: 05/22/2023]
Abstract
Cardiac magnetic resonance (CMR) image segmentation is an integral step in the analysis of cardiac function and diagnosis of heart related diseases. While recent deep learning-based approaches in automatic segmentation have shown great promise to alleviate the need for manual segmentation, most of these are not applicable to realistic clinical scenarios. This is largely due to training on mainly homogeneous datasets, without variation in acquisition, which typically occurs in multi-vendor and multi-site settings, as well as pathological data. Such approaches frequently exhibit a degradation in prediction performance, particularly on outlier cases commonly associated with difficult pathologies, artifacts and extensive changes in tissue shape and appearance. In this work, we present a model aimed at segmenting all three cardiac structures in a multi-center, multi-disease and multi-view scenario. We propose a pipeline, addressing different challenges with segmentation of such heterogeneous data, consisting of heart region detection, augmentation through image synthesis and a late-fusion segmentation approach. Extensive experiments and analysis demonstrate the ability of the proposed approach to tackle the presence of outlier cases during both training and testing, allowing for better adaptation to unseen and difficult examples. Overall, we show that the effective reduction of segmentation failures on outlier cases has a positive impact on not only the average segmentation performance, but also on the estimation of clinical parameters, leading to a better consistency in derived metrics.
Collapse
Affiliation(s)
- Yasmina Al Khalil
- Department of Biomedical Engineering, Eindhoven University of Technology, Eindhoven, The Netherlands.
| | - Sina Amirrajab
- Department of Biomedical Engineering, Eindhoven University of Technology, Eindhoven, The Netherlands
| | | | | | - Josien Pluim
- Department of Biomedical Engineering, Eindhoven University of Technology, Eindhoven, The Netherlands
| | - Marcel Breeuwer
- Department of Biomedical Engineering, Eindhoven University of Technology, Eindhoven, The Netherlands; Philips Healthcare, MR R&D - Clinical Science, Best, The Netherlands
| |
Collapse
|
36
|
Rezaei SR, Ahmadi A. A GAN-based method for 3D lung tumor reconstruction boosted by a knowledge transfer approach. MULTIMEDIA TOOLS AND APPLICATIONS 2023:1-27. [PMID: 37362675 PMCID: PMC10106883 DOI: 10.1007/s11042-023-15232-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 05/05/2022] [Revised: 02/18/2023] [Accepted: 03/30/2023] [Indexed: 06/28/2023]
Abstract
Three-dimensional (3D) image reconstruction of tumors has been one of the most effective techniques for accurately visualizing tumor structures and treatment with high resolution, which requires a set of two-dimensional medical images such as CT slices. In this paper we propose a novel method based on generative adversarial networks (GANs) for 3D lung tumor reconstruction by CT images. The proposed method consists of three stages: lung segmentation, tumor segmentation and 3D lung tumor reconstruction. Lung and tumor segmentation are performed using snake optimization and Gustafson-Kessel (GK) clustering. In the 3D reconstruction part first, features are extracted using the pre-trained VGG model from the tumors that detected in 2D CT slices. Then, a sequence of extracted features is fed into an LSTM to output compressed features. Finally, the compressed feature is used as input for GAN, where the generator is responsible for high-level reconstructing the 3D image of the lung tumor. The main novelty of this paper is the use of GAN to reconstruct a 3D lung tumor model for the first time, to the best of our knowledge. Also, we used knowledge transfer to extract features from 2D images to speed up the training process. The results obtained from the proposed model on the LUNA dataset showed better results than state of the art. According to HD and ED metrics, the proposed method has the lowest values of 3.02 and 1.06, respectively, as compared to those of other methods. The experimental results show that the proposed method performs better than previous similar methods and it is useful to help practitioners in the treatment process.
Collapse
Affiliation(s)
- Seyed Reza Rezaei
- Department of Industrial Engineering and Management Systems, Amirkabir University of Technology, Tehran, Iran
| | - Abbas Ahmadi
- Department of Industrial Engineering and Management Systems, Amirkabir University of Technology, Tehran, Iran
| |
Collapse
|
37
|
Li L, Wu F, Wang S, Luo X, Martín-Isla C, Zhai S, Zhang J, Liu Y, Zhang Z, Ankenbrand MJ, Jiang H, Zhang X, Wang L, Arega TW, Altunok E, Zhao Z, Li F, Ma J, Yang X, Puybareau E, Oksuz I, Bricq S, Li W, Punithakumar K, Tsaftaris SA, Schreiber LM, Yang M, Liu G, Xia Y, Wang G, Escalera S, Zhuang X. MyoPS: A benchmark of myocardial pathology segmentation combining three-sequence cardiac magnetic resonance images. Med Image Anal 2023; 87:102808. [PMID: 37087838 DOI: 10.1016/j.media.2023.102808] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/08/2022] [Revised: 01/11/2023] [Accepted: 03/30/2023] [Indexed: 04/07/2023]
Abstract
Assessment of myocardial viability is essential in diagnosis and treatment management of patients suffering from myocardial infarction, and classification of pathology on the myocardium is the key to this assessment. This work defines a new task of medical image analysis, i.e., to perform myocardial pathology segmentation (MyoPS) combining three-sequence cardiac magnetic resonance (CMR) images, which was first proposed in the MyoPS challenge, in conjunction with MICCAI 2020. Note that MyoPS refers to both myocardial pathology segmentation and the challenge in this paper. The challenge provided 45 paired and pre-aligned CMR images, allowing algorithms to combine the complementary information from the three CMR sequences for pathology segmentation. In this article, we provide details of the challenge, survey the works from fifteen participants and interpret their methods according to five aspects, i.e., preprocessing, data augmentation, learning strategy, model architecture and post-processing. In addition, we analyze the results with respect to different factors, in order to examine the key obstacles and explore the potential of solutions, as well as to provide a benchmark for future research. The average Dice scores of submitted algorithms were 0.614±0.231 and 0.644±0.153 for myocardial scars and edema, respectively. We conclude that while promising results have been reported, the research is still in the early stage, and more in-depth exploration is needed before a successful application to the clinics. MyoPS data and evaluation tool continue to be publicly available upon registration via its homepage (www.sdspeople.fudan.edu.cn/zhuangxiahai/0/myops20/).
Collapse
Affiliation(s)
- Lei Li
- School of Data Science, Fudan University, Shanghai, China; School of Biomedical Engineering, Shanghai Jiao Tong University, Shanghai, China.
| | - Fuping Wu
- School of Data Science, Fudan University, Shanghai, China.
| | - Sihan Wang
- School of Data Science, Fudan University, Shanghai, China.
| | - Xinzhe Luo
- School of Data Science, Fudan University, Shanghai, China
| | - Carlos Martín-Isla
- Departament de Matemàtiques & Informàtica, Universitat de Barcelona, Barcelona, Spain
| | - Shuwei Zhai
- School of Mechanical and Electrical Engineering, University of Electronic Science and Technology of China, Chengdu, China
| | - Jianpeng Zhang
- School of Computer Science and Engineering, Northwestern Polytechnical University, Xi'an, China
| | - Yanfei Liu
- College of Electrical and Information Engineering, Hunan University, Changsha, China
| | - Zhen Zhang
- College of Physics and Information Engineering, Fuzhou University, Fuzhou, China
| | - Markus J Ankenbrand
- Chair of Molecular and Cellular Imaging, Comprehensive Heart Failure Center, Wuerzburg University Hospitals, Wuerzburg, Germany
| | - Haochuan Jiang
- School of Engineering, University of Edinburgh, Edinburgh, UK; School of Robotics, Xi'an Jiaotong-Liverpool University, Suzhou, China
| | - Xiaoran Zhang
- Department of Electrical and Computer Engineering, University of California, LA, USA
| | - Linhong Wang
- Chongqing Key Laboratory of Image Cognition, Chongqing University of Posts and Telecommunications, Chongqing, China
| | | | - Elif Altunok
- Computer Engineering Department, Istanbul Technical University, Istanbul, Turkey
| | - Zhou Zhao
- EPITA Research and Development Laboratory (LRDE), Le Kremlin-Bicêtre, France
| | - Feiyan Li
- Chongqing Key Laboratory of Image Cognition, Chongqing University of Posts and Telecommunications, Chongqing, China
| | - Jun Ma
- Department of Mathematics, Nanjing University of Science and Technology, Nanjing, China
| | - Xiaoping Yang
- Department of Mathematics, Nanjing University, Nanjing, China
| | - Elodie Puybareau
- EPITA Research and Development Laboratory (LRDE), Le Kremlin-Bicêtre, France
| | - Ilkay Oksuz
- Computer Engineering Department, Istanbul Technical University, Istanbul, Turkey
| | - Stephanie Bricq
- ImViA Laboratory, Université Bourgogne Franche-Comté, Dijon, France
| | - Weisheng Li
- Chongqing Key Laboratory of Image Cognition, Chongqing University of Posts and Telecommunications, Chongqing, China
| | | | | | - Laura M Schreiber
- Chair of Molecular and Cellular Imaging, Comprehensive Heart Failure Center, Wuerzburg University Hospitals, Wuerzburg, Germany
| | - Mingjing Yang
- College of Physics and Information Engineering, Fuzhou University, Fuzhou, China
| | - Guocai Liu
- College of Electrical and Information Engineering, Hunan University, Changsha, China; National Engineering Laboratory for Robot Visual Perception and Control Technology, Changsha, China
| | - Yong Xia
- School of Computer Science and Engineering, Northwestern Polytechnical University, Xi'an, China
| | - Guotai Wang
- School of Mechanical and Electrical Engineering, University of Electronic Science and Technology of China, Chengdu, China
| | - Sergio Escalera
- Departament de Matemàtiques & Informàtica, Universitat de Barcelona, Barcelona, Spain; Computer Vision Center, Universitat Autònoma de Barcelona, Spain
| | - Xiahai Zhuang
- School of Data Science, Fudan University, Shanghai, China.
| |
Collapse
|
38
|
Li K, Yu L, Heng PA. Domain-Incremental Cardiac Image Segmentation With Style-Oriented Replay and Domain-Sensitive Feature Whitening. IEEE TRANSACTIONS ON MEDICAL IMAGING 2023; 42:570-581. [PMID: 36191115 DOI: 10.1109/tmi.2022.3211195] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/16/2023]
Abstract
Contemporary methods have shown promising results on cardiac image segmentation, but merely in static learning, i.e., optimizing the network once for all, ignoring potential needs for model updating. In real-world scenarios, new data continues to be gathered from multiple institutions over time and new demands keep growing to pursue more satisfying performance. The desired model should incrementally learn from each incoming dataset and progressively update with improved functionality as time goes by. As the datasets sequentially delivered from multiple sites are normally heterogenous with domain discrepancy, each updated model should not catastrophically forget previously learned domains while well generalizing to currently arrived domains or even unseen domains. In medical scenarios, this is particularly challenging as accessing or storing past data is commonly not allowed due to data privacy. To this end, we propose a novel domain-incremental learning framework to recover past domain inputs first and then regularly replay them during model optimization. Particularly, we first present a style-oriented replay module to enable structure-realistic and memory-efficient reproduction of past data, and then incorporate the replayed past data to jointly optimize the model with current data to alleviate catastrophic forgetting. During optimization, we additionally perform domain-sensitive feature whitening to suppress model's dependency on features that are sensitive to domain changes (e.g., domain-distinctive style features) to assist domain-invariant feature exploration and gradually improve the generalization performance of the network. We have extensively evaluated our approach with the M&Ms Dataset in single-domain and compound-domain incremental learning settings. Our approach outperforms other comparison methods with less forgetting on past domains and better generalization on current domains and unseen domains.
Collapse
|
39
|
Schwarz EL, Pegolotti L, Pfaller MR, Marsden AL. Beyond CFD: Emerging methodologies for predictive simulation in cardiovascular health and disease. BIOPHYSICS REVIEWS 2023; 4:011301. [PMID: 36686891 PMCID: PMC9846834 DOI: 10.1063/5.0109400] [Citation(s) in RCA: 6] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/12/2022] [Accepted: 12/12/2022] [Indexed: 01/15/2023]
Abstract
Physics-based computational models of the cardiovascular system are increasingly used to simulate hemodynamics, tissue mechanics, and physiology in evolving healthy and diseased states. While predictive models using computational fluid dynamics (CFD) originated primarily for use in surgical planning, their application now extends well beyond this purpose. In this review, we describe an increasingly wide range of modeling applications aimed at uncovering fundamental mechanisms of disease progression and development, performing model-guided design, and generating testable hypotheses to drive targeted experiments. Increasingly, models are incorporating multiple physical processes spanning a wide range of time and length scales in the heart and vasculature. With these expanded capabilities, clinical adoption of patient-specific modeling in congenital and acquired cardiovascular disease is also increasing, impacting clinical care and treatment decisions in complex congenital heart disease, coronary artery disease, vascular surgery, pulmonary artery disease, and medical device design. In support of these efforts, we discuss recent advances in modeling methodology, which are most impactful when driven by clinical needs. We describe pivotal recent developments in image processing, fluid-structure interaction, modeling under uncertainty, and reduced order modeling to enable simulations in clinically relevant timeframes. In all these areas, we argue that traditional CFD alone is insufficient to tackle increasingly complex clinical and biological problems across scales and systems. Rather, CFD should be coupled with appropriate multiscale biological, physical, and physiological models needed to produce comprehensive, impactful models of mechanobiological systems and complex clinical scenarios. With this perspective, we finally outline open problems and future challenges in the field.
Collapse
Affiliation(s)
- Erica L. Schwarz
- Departments of Pediatrics and Bioengineering, Stanford University, Stanford, California 94305, USA
| | - Luca Pegolotti
- Departments of Pediatrics and Bioengineering, Stanford University, Stanford, California 94305, USA
| | - Martin R. Pfaller
- Departments of Pediatrics and Bioengineering, Stanford University, Stanford, California 94305, USA
| | - Alison L. Marsden
- Departments of Pediatrics and Bioengineering, Stanford University, Stanford, California 94305, USA
| |
Collapse
|
40
|
Billot B, Greve DN, Puonti O, Thielscher A, Van Leemput K, Fischl B, Dalca AV, Iglesias JE. SynthSeg: Segmentation of brain MRI scans of any contrast and resolution without retraining. Med Image Anal 2023; 86:102789. [PMID: 36857946 PMCID: PMC10154424 DOI: 10.1016/j.media.2023.102789] [Citation(s) in RCA: 51] [Impact Index Per Article: 51.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/26/2022] [Revised: 01/20/2023] [Accepted: 02/22/2023] [Indexed: 03/03/2023]
Abstract
Despite advances in data augmentation and transfer learning, convolutional neural networks (CNNs) difficultly generalise to unseen domains. When segmenting brain scans, CNNs are highly sensitive to changes in resolution and contrast: even within the same MRI modality, performance can decrease across datasets. Here we introduce SynthSeg, the first segmentation CNN robust against changes in contrast and resolution. SynthSeg is trained with synthetic data sampled from a generative model conditioned on segmentations. Crucially, we adopt a domain randomisation strategy where we fully randomise the contrast and resolution of the synthetic training data. Consequently, SynthSeg can segment real scans from a wide range of target domains without retraining or fine-tuning, which enables straightforward analysis of huge amounts of heterogeneous clinical data. Because SynthSeg only requires segmentations to be trained (no images), it can learn from labels obtained by automated methods on diverse populations (e.g., ageing and diseased), thus achieving robustness to a wide range of morphological variability. We demonstrate SynthSeg on 5,000 scans of six modalities (including CT) and ten resolutions, where it exhibits unparallelled generalisation compared with supervised CNNs, state-of-the-art domain adaptation, and Bayesian segmentation. Finally, we demonstrate the generalisability of SynthSeg by applying it to cardiac MRI and CT scans.
Collapse
Affiliation(s)
- Benjamin Billot
- Centre for Medical Image Computing, University College London, UK.
| | - Douglas N Greve
- Martinos Center for Biomedical Imaging, Department of Radiology, Massachusetts General Hospital and Harvard Medical School, USA
| | - Oula Puonti
- Danish Research Centre for Magnetic Resonance, Centre for Functional and Diagnostic Imaging and Research, Copenhagen University Hospital, Denmark
| | - Axel Thielscher
- Danish Research Centre for Magnetic Resonance, Centre for Functional and Diagnostic Imaging and Research, Copenhagen University Hospital, Denmark; Department of Health Technology, Technical University of, Denmark
| | - Koen Van Leemput
- Martinos Center for Biomedical Imaging, Department of Radiology, Massachusetts General Hospital and Harvard Medical School, USA; Department of Health Technology, Technical University of, Denmark
| | - Bruce Fischl
- Martinos Center for Biomedical Imaging, Department of Radiology, Massachusetts General Hospital and Harvard Medical School, USA; Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, USA; Program in Health Sciences and Technology, Massachusetts Institute of Technology, USA
| | - Adrian V Dalca
- Martinos Center for Biomedical Imaging, Department of Radiology, Massachusetts General Hospital and Harvard Medical School, USA; Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, USA
| | - Juan Eugenio Iglesias
- Centre for Medical Image Computing, University College London, UK; Martinos Center for Biomedical Imaging, Department of Radiology, Massachusetts General Hospital and Harvard Medical School, USA; Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, USA
| | | |
Collapse
|
41
|
Kong F, Shadden SC. Learning Whole Heart Mesh Generation From Patient Images for Computational Simulations. IEEE TRANSACTIONS ON MEDICAL IMAGING 2023; 42:533-545. [PMID: 36327186 DOI: 10.1109/tmi.2022.3219284] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/16/2023]
Abstract
Patient-specific cardiac modeling combines geometries of the heart derived from medical images and biophysical simulations to predict various aspects of cardiac function. However, generating simulation-suitable models of the heart from patient image data often requires complicated procedures and significant human effort. We present a fast and automated deep-learning method to construct simulation-suitable models of the heart from medical images. The approach constructs meshes from 3D patient images by learning to deform a small set of deformation handles on a whole heart template. For both 3D CT and MR data, this method achieves promising accuracy for whole heart reconstruction, consistently outperforming prior methods in constructing simulation-suitable meshes of the heart. When evaluated on time-series CT data, this method produced more anatomically and temporally consistent geometries than prior methods, and was able to produce geometries that better satisfy modeling requirements for cardiac flow simulations. Our source code and pretrained networks are available at https://github.com/fkong7/HeartDeformNets.
Collapse
|
42
|
Qiu J, Li L, Wang S, Zhang K, Chen Y, Yang S, Zhuang X. MyoPS-Net: Myocardial pathology segmentation with flexible combination of multi-sequence CMR images. Med Image Anal 2023; 84:102694. [PMID: 36495601 DOI: 10.1016/j.media.2022.102694] [Citation(s) in RCA: 5] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/02/2022] [Revised: 10/05/2022] [Accepted: 11/16/2022] [Indexed: 11/29/2022]
Abstract
Myocardial pathology segmentation (MyoPS) can be a prerequisite for the accurate diagnosis and treatment planning of myocardial infarction. However, achieving this segmentation is challenging, mainly due to the inadequate and indistinct information from an image. In this work, we develop an end-to-end deep neural network, referred to as MyoPS-Net, to flexibly combine five-sequence cardiac magnetic resonance (CMR) images for MyoPS. To extract precise and adequate information, we design an effective yet flexible architecture to extract and fuse cross-modal features. This architecture can tackle different numbers of CMR images and complex combinations of modalities, with output branches targeting specific pathologies. To impose anatomical knowledge on the segmentation results, we first propose a module to regularize myocardium consistency and localize the pathologies, and then introduce an inclusiveness loss to utilize relations between myocardial scars and edema. We evaluated the proposed MyoPS-Net on two datasets, i.e., a private one consisting of 50 paired multi-sequence CMR images and a public one from MICCAI2020 MyoPS Challenge. Experimental results showed that MyoPS-Net could achieve state-of-the-art performance in various scenarios. Note that in practical clinics, the subjects may not have full sequences, such as missing LGE CMR or mapping CMR scans. We therefore conducted extensive experiments to investigate the performance of the proposed method in dealing with such complex combinations of different CMR sequences. Results proved the superiority and generalizability of MyoPS-Net, and more importantly, indicated a practical clinical application. The code has been released via https://github.com/QJYBall/MyoPS-Net.
Collapse
Affiliation(s)
- Junyi Qiu
- School of Data Science, Fudan University, Shanghai, China
| | - Lei Li
- Institute of Biomedical Engineering, University of Oxford, Oxford, UK
| | - Sihan Wang
- School of Data Science, Fudan University, Shanghai, China
| | - Ke Zhang
- School of Data Science, Fudan University, Shanghai, China
| | - Yinyin Chen
- Department of Radiology, Zhongshan Hospital, Fudan University, Shanghai, China; Department of Medical Imaging, Shanghai Medical School, Fudan University and Shanghai Institute of Medical Imaging, Shanghai, China
| | - Shan Yang
- Department of Radiology, Zhongshan Hospital, Fudan University, Shanghai, China; Department of Medical Imaging, Shanghai Medical School, Fudan University and Shanghai Institute of Medical Imaging, Shanghai, China
| | - Xiahai Zhuang
- School of Data Science, Fudan University, Shanghai, China.
| |
Collapse
|
43
|
Wang X, Wang F, Niu Y. Two-Stage CNN Whole Heart Segmentation Combining Image Enhanced Attention Mechanism and Metric Classification. J Digit Imaging 2023; 36:124-142. [PMID: 36175600 PMCID: PMC9984643 DOI: 10.1007/s10278-022-00708-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/30/2021] [Revised: 09/06/2022] [Accepted: 09/17/2022] [Indexed: 10/14/2022] Open
Abstract
Accurate segmentation of multiple tissues and organs in cardiac medical imaging is of great value in computer-aided cardiovascular diagnosis. However, it is challenging due to the complex distribution of various tissues and organs in cardiac MRI (magnetic resonance imaging) slices, low discriminative and large spanning organs. To handle these problems, a two-stage CNN (convolutional neural network) segmentation method based on the combination of Log-Gabor filter attention mechanism and metric classification is proposed. The Log-Gabor filterbank is applied to selectively enhance the texture information and contour information of each tissue and organ, and the spatial and channel attention mechanism jointly with the varying kernel size of Log-Gabor filterbank is incorporated into the codec structure to adaptively extract target features of different sizes and focus on the discriminative features in the network. To solve the problem of insufficient segmentation on subtle and adherent edges involving different tissues, a metric classification network is incorporated to finely optimize the hard-to-be-segmented boundaries. The proposed method was tested on cardiac MRI data set to segment 7 cardiac tissues, and the rationality and effectiveness of the method were verified. In comparison to a series of deep learning-based segmentation models, the proposed method achieves competitive performance.
Collapse
Affiliation(s)
- Xuchu Wang
- Key Laboratory of Optoelectronic Technology and Systems of Ministry of Education, College of Optoelectronic Engineering, Chongqing University, Chongqing, 400040, China.
| | - Fusheng Wang
- College of Optoelectronic Engineering, Chongqing University, Chongqing, 400040, China
| | - Yanmin Niu
- College of Computer and Information Science, Chongqing Normal University, Chongqing, 400050, China
| |
Collapse
|
44
|
Byrne N, Clough JR, Valverde I, Montana G, King AP. A Persistent Homology-Based Topological Loss for CNN-Based Multiclass Segmentation of CMR. IEEE TRANSACTIONS ON MEDICAL IMAGING 2023; 42:3-14. [PMID: 36044487 PMCID: PMC7614102 DOI: 10.1109/tmi.2022.3203309] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/10/2023]
Abstract
Multi-class segmentation of cardiac magnetic resonance (CMR) images seeks a separation of data into anatomical components with known structure and configuration. The most popular CNN-based methods are optimised using pixel wise loss functions, ignorant of the spatially extended features that characterise anatomy. Therefore, whilst sharing a high spatial overlap with the ground truth, inferred CNN-based segmentations can lack coherence, including spurious connected components, holes and voids. Such results are implausible, violating anticipated anatomical topology. In response, (single-class) persistent homology-based loss functions have been proposed to capture global anatomical features. Our work extends these approaches to the task of multi-class segmentation. Building an enriched topological description of all class labels and class label pairs, our loss functions make predictable and statistically significant improvements in segmentation topology using a CNN-based post-processing framework. We also present (and make available) a highly efficient implementation based on cubical complexes and parallel execution, enabling practical application within high resolution 3D data for the first time. We demonstrate our approach on 2D short axis and 3D whole heart CMR segmentation, advancing a detailed and faithful analysis of performance on two publicly available datasets.
Collapse
Affiliation(s)
- Nick Byrne
- School of BMEIS at KCL: London, SE1 7EH, UK. Nick Byrne and Isra Valverde are also with the Medical Physics and Paediatric Cardiology Departments respectively, both at GSTT: London, SE1 7EH, UK
| | - James R. Clough
- School of BMEIS at KCL: London, SE1 7EH, UK. Nick Byrne and Isra Valverde are also with the Medical Physics and Paediatric Cardiology Departments respectively, both at GSTT: London, SE1 7EH, UK
| | - Israel Valverde
- School of BMEIS at KCL: London, SE1 7EH, UK. Nick Byrne and Isra Valverde are also with the Medical Physics and Paediatric Cardiology Departments respectively, both at GSTT: London, SE1 7EH, UK
| | - Giovanni Montana
- Warwick Manufacturing Group at the University of Warwick: Coventry, CV4 7AL, UK
| | - Andrew P. King
- School of BMEIS at KCL: London, SE1 7EH, UK. Nick Byrne and Isra Valverde are also with the Medical Physics and Paediatric Cardiology Departments respectively, both at GSTT: London, SE1 7EH, UK
| |
Collapse
|
45
|
Dorent R, Kujawa A, Ivory M, Bakas S, Rieke N, Joutard S, Glocker B, Cardoso J, Modat M, Batmanghelich K, Belkov A, Calisto MB, Choi JW, Dawant BM, Dong H, Escalera S, Fan Y, Hansen L, Heinrich MP, Joshi S, Kashtanova V, Kim HG, Kondo S, Kruse CN, Lai-Yuen SK, Li H, Liu H, Ly B, Oguz I, Shin H, Shirokikh B, Su Z, Wang G, Wu J, Xu Y, Yao K, Zhang L, Ourselin S, Shapey J, Vercauteren T. CrossMoDA 2021 challenge: Benchmark of cross-modality domain adaptation techniques for vestibular schwannoma and cochlea segmentation. Med Image Anal 2023; 83:102628. [PMID: 36283200 DOI: 10.1016/j.media.2022.102628] [Citation(s) in RCA: 8] [Impact Index Per Article: 8.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/20/2021] [Revised: 06/17/2022] [Accepted: 09/10/2022] [Indexed: 02/04/2023]
Abstract
Domain Adaptation (DA) has recently been of strong interest in the medical imaging community. While a large variety of DA techniques have been proposed for image segmentation, most of these techniques have been validated either on private datasets or on small publicly available datasets. Moreover, these datasets mostly addressed single-class problems. To tackle these limitations, the Cross-Modality Domain Adaptation (crossMoDA) challenge was organised in conjunction with the 24th International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI 2021). CrossMoDA is the first large and multi-class benchmark for unsupervised cross-modality Domain Adaptation. The goal of the challenge is to segment two key brain structures involved in the follow-up and treatment planning of vestibular schwannoma (VS): the VS and the cochleas. Currently, the diagnosis and surveillance in patients with VS are commonly performed using contrast-enhanced T1 (ceT1) MR imaging. However, there is growing interest in using non-contrast imaging sequences such as high-resolution T2 (hrT2) imaging. For this reason, we established an unsupervised cross-modality segmentation benchmark. The training dataset provides annotated ceT1 scans (N=105) and unpaired non-annotated hrT2 scans (N=105). The aim was to automatically perform unilateral VS and bilateral cochlea segmentation on hrT2 scans as provided in the testing set (N=137). This problem is particularly challenging given the large intensity distribution gap across the modalities and the small volume of the structures. A total of 55 teams from 16 countries submitted predictions to the validation leaderboard. Among them, 16 teams from 9 different countries submitted their algorithm for the evaluation phase. The level of performance reached by the top-performing teams is strikingly high (best median Dice score - VS: 88.4%; Cochleas: 85.7%) and close to full supervision (median Dice score - VS: 92.5%; Cochleas: 87.7%). All top-performing methods made use of an image-to-image translation approach to transform the source-domain images into pseudo-target-domain images. A segmentation network was then trained using these generated images and the manual annotations provided for the source image.
Collapse
Affiliation(s)
- Reuben Dorent
- School of Biomedical Engineering & Imaging Sciences, King's College London, London, United Kingdom.
| | - Aaron Kujawa
- School of Biomedical Engineering & Imaging Sciences, King's College London, London, United Kingdom
| | - Marina Ivory
- School of Biomedical Engineering & Imaging Sciences, King's College London, London, United Kingdom
| | - Spyridon Bakas
- Center for Biomedical Image Computing and Analytics (CBICA), University of Pennsylvania, Philadelphia, USA; Department of Pathology and Laboratory Medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA; Department of Radiology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
| | | | - Samuel Joutard
- School of Biomedical Engineering & Imaging Sciences, King's College London, London, United Kingdom
| | - Ben Glocker
- Department of Computing, Imperial College London, Department of Computing, London, United Kingdom
| | - Jorge Cardoso
- School of Biomedical Engineering & Imaging Sciences, King's College London, London, United Kingdom
| | - Marc Modat
- School of Biomedical Engineering & Imaging Sciences, King's College London, London, United Kingdom
| | | | - Arseniy Belkov
- Moscow Institute of Physics and Technology, Moscow, Russia
| | | | - Jae Won Choi
- Department of Radiology, Armed Forces Yangju Hospital, Yangju, Republic of Korea
| | | | - Hexin Dong
- Center for Data Science, Peking University, Beijing, China
| | - Sergio Escalera
- Artificial Intelligence in Medicine Lab (BCN-AIM) and Human Behavior Analysis Lab (HuPBA), Universitat de Barcelona, Barcelona, Spain
| | - Yubo Fan
- Vanderbilt University, Nashville, USA
| | - Lasse Hansen
- Institute of Medical Informatics, Universität zu Lübeck, Germany
| | | | - Smriti Joshi
- Artificial Intelligence in Medicine Lab (BCN-AIM) and Human Behavior Analysis Lab (HuPBA), Universitat de Barcelona, Barcelona, Spain
| | | | - Hyeon Gyu Kim
- School of Electrical and Electronic Engineering, Yonsei University, Seoul, Republic of Korea
| | | | | | | | - Hao Li
- Vanderbilt University, Nashville, USA
| | - Han Liu
- Vanderbilt University, Nashville, USA
| | - Buntheng Ly
- Inria, Université Côte d'Azur, Sophia Antipolis, France
| | - Ipek Oguz
- Vanderbilt University, Nashville, USA
| | - Hyungseob Shin
- School of Electrical and Electronic Engineering, Yonsei University, Seoul, Republic of Korea
| | - Boris Shirokikh
- Skolkovo Institute of Science and Technology, Moscow, Russia; Artificial Intelligence Research Institute (AIRI), Moscow, Russia
| | - Zixian Su
- University of Liverpool, Liverpool, United Kingdom; School of Advanced Technology, Xi'an Jiaotong-Liverpool University, Suzhou, China
| | - Guotai Wang
- School of Mechanical and Electrical Engineering, University of Electronic Science and Technology of China, Chengdu, China
| | - Jianghao Wu
- School of Mechanical and Electrical Engineering, University of Electronic Science and Technology of China, Chengdu, China
| | - Yanwu Xu
- Department of Biomedical Informatics, University of Pittsburgh, Pittsburgh, USA
| | - Kai Yao
- University of Liverpool, Liverpool, United Kingdom; School of Advanced Technology, Xi'an Jiaotong-Liverpool University, Suzhou, China
| | - Li Zhang
- Center for Data Science, Peking University, Beijing, China
| | - Sébastien Ourselin
- School of Biomedical Engineering & Imaging Sciences, King's College London, London, United Kingdom
| | - Jonathan Shapey
- School of Biomedical Engineering & Imaging Sciences, King's College London, London, United Kingdom; Department of Neurosurgery, King's College Hospital, London, United Kingdom
| | - Tom Vercauteren
- School of Biomedical Engineering & Imaging Sciences, King's College London, London, United Kingdom
| |
Collapse
|
46
|
Chen X, Peng Y, Guo Y, Sun J, Li D, Cui J. MLRD-Net: 3D multiscale local cross-channel residual denoising network for MRI-based brain tumor segmentation. Med Biol Eng Comput 2022; 60:3377-3395. [DOI: 10.1007/s11517-022-02673-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/26/2021] [Accepted: 09/17/2022] [Indexed: 11/11/2022]
|
47
|
MRDFF: A deep forest based framework for CT whole heart segmentation. Methods 2022; 208:48-58. [DOI: 10.1016/j.ymeth.2022.10.005] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/20/2022] [Revised: 06/27/2022] [Accepted: 10/18/2022] [Indexed: 11/06/2022] Open
|
48
|
Bui V, Hsu LY, Chang LC, Sun AY, Tran L, Shanbhag SM, Zhou W, Mehta NN, Chen MY. DeepHeartCT: A fully automatic artificial intelligence hybrid framework based on convolutional neural network and multi-atlas segmentation for multi-structure cardiac computed tomography angiography image segmentation. Front Artif Intell 2022; 5:1059007. [PMID: 36483981 PMCID: PMC9723331 DOI: 10.3389/frai.2022.1059007] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/30/2022] [Accepted: 11/03/2022] [Indexed: 01/25/2023] Open
Abstract
Cardiac computed tomography angiography (CTA) is an emerging imaging modality for assessing coronary artery as well as various cardiovascular structures. Recently, deep learning (DL) methods have been successfully applied to many applications of medical image analysis including cardiac CTA structure segmentation. However, DL requires a large amounts of data and high-quality labels for training which can be burdensome to obtain due to its labor-intensive nature. In this study, we aim to develop a fully automatic artificial intelligence (AI) system, named DeepHeartCT, for accurate and rapid cardiac CTA segmentation based on DL. The proposed system was trained using a large clinical dataset with computer-generated labels to segment various cardiovascular structures including left and right ventricles (LV, RV), left and right atria (LA, RA), and LV myocardium (LVM). This new system was trained directly using high-quality computer labels generated from our previously developed multi-atlas based AI system. In addition, a reverse ranking strategy was proposed to assess the segmentation quality in the absence of manual reference labels. This strategy allowed the new framework to assemble optimal computer-generated labels from a large dataset for effective training of a deep convolutional neural network (CNN). A large clinical cardiac CTA studies (n = 1,064) were used to train and validate our framework. The trained model was then tested on another independent dataset with manual labels (n = 60). The Dice score, Hausdorff distance and mean surface distance were used to quantify the segmentation accuracy. The proposed DeepHeartCT framework yields a high median Dice score of 0.90 [interquartile range (IQR), 0.90-0.91], a low median Hausdorff distance of 7 mm (IQR, 4-15 mm) and a low mean surface distance of 0.80 mm (IQR, 0.57-1.29 mm) across all segmented structures. An additional experiment was conducted to evaluate the proposed DL-based AI framework trained with a small vs. large dataset. The results show our framework also performed well when trained on a small optimal training dataset (n = 110) with a significantly reduced training time. These results demonstrated that the proposed DeepHeartCT framework provides accurate and rapid cardiac CTA segmentation that can be readily generalized for handling large-scale medical imaging applications.
Collapse
Affiliation(s)
- Vy Bui
- National Heart Lung and Blood Institute, National Institutes of Health, Bethesda, MD, United States
| | - Li-Yueh Hsu
- Radiology and Imaging Sciences, Clinical Center, National Institutes of Health, Bethesda, MD, United States,*Correspondence: Li-Yueh Hsu
| | - Lin-Ching Chang
- Department of Electrical Engineering and Computer Science, Catholic University of America, Washington, DC, United States
| | - An-Yu Sun
- National Heart Lung and Blood Institute, National Institutes of Health, Bethesda, MD, United States,Department of Electrical Engineering and Computer Science, Catholic University of America, Washington, DC, United States
| | - Loc Tran
- National Heart Lung and Blood Institute, National Institutes of Health, Bethesda, MD, United States,Department of Electrical Engineering and Computer Science, Catholic University of America, Washington, DC, United States
| | - Sujata M. Shanbhag
- National Heart Lung and Blood Institute, National Institutes of Health, Bethesda, MD, United States
| | - Wunan Zhou
- National Heart Lung and Blood Institute, National Institutes of Health, Bethesda, MD, United States
| | - Nehal N. Mehta
- National Heart Lung and Blood Institute, National Institutes of Health, Bethesda, MD, United States
| | - Marcus Y. Chen
- National Heart Lung and Blood Institute, National Institutes of Health, Bethesda, MD, United States
| |
Collapse
|
49
|
Montalt-Tordera J, Pajaziti E, Jones R, Sauvage E, Puranik R, Singh AAV, Capelli C, Steeden J, Schievano S, Muthurangu V. Automatic segmentation of the great arteries for computational hemodynamic assessment. J Cardiovasc Magn Reson 2022; 24:57. [PMID: 36336682 PMCID: PMC9639271 DOI: 10.1186/s12968-022-00891-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/29/2021] [Accepted: 10/03/2022] [Indexed: 11/09/2022] Open
Abstract
BACKGROUND Computational fluid dynamics (CFD) is increasingly used for the assessment of blood flow conditions in patients with congenital heart disease (CHD). This requires patient-specific anatomy, typically obtained from segmented 3D cardiovascular magnetic resonance (CMR) images. However, segmentation is time-consuming and requires expert input. This study aims to develop and validate a machine learning (ML) method for segmentation of the aorta and pulmonary arteries for CFD studies. METHODS 90 CHD patients were retrospectively selected for this study. 3D CMR images were manually segmented to obtain ground-truth (GT) background, aorta and pulmonary artery labels. These were used to train and optimize a U-Net model, using a 70-10-10 train-validation-test split. Segmentation performance was primarily evaluated using Dice score. CFD simulations were set up from GT and ML segmentations using a semi-automatic meshing and simulation pipeline. Mean pressure and velocity fields across 99 planes along the vessel centrelines were extracted, and a mean average percentage error (MAPE) was calculated for each vessel pair (ML vs GT). A second observer (SO) segmented the test dataset for assessment of inter-observer variability. Friedman tests were used to compare ML vs GT, SO vs GT and ML vs SO metrics, and pressure/velocity field errors. RESULTS The network's Dice score (ML vs GT) was 0.945 (interquartile range: 0.929-0.955) for the aorta and 0.885 (0.851-0.899) for the pulmonary arteries. Differences with the inter-observer Dice score (SO vs GT) and ML vs SO Dice scores were not statistically significant for either aorta or pulmonary arteries (p = 0.741, p = 0.061). The ML vs GT MAPEs for pressure and velocity in the aorta were 10.1% (8.5-15.7%) and 4.1% (3.1-6.9%), respectively, and for the pulmonary arteries 14.6% (11.5-23.2%) and 6.3% (4.3-7.9%), respectively. Inter-observer (SO vs GT) and ML vs SO pressure and velocity MAPEs were of a similar magnitude to ML vs GT (p > 0.2). CONCLUSIONS ML can successfully segment the great vessels for CFD, with errors similar to inter-observer variability. This fast, automatic method reduces the time and effort needed for CFD analysis, making it more attractive for routine clinical use.
Collapse
Affiliation(s)
| | | | - Rod Jones
- Great Ormond Street Hospital, London, UK
| | - Emilie Sauvage
- UCL Institute of Cardiovascular Science, UCL, London, UK
| | - Rajesh Puranik
- Children’s Hospital at Westmead, Sydney, Australia
- Faculty of Medicine and Health, University of Sydney, Sydney, Australia
| | - Aakansha Ajay Vir Singh
- Children’s Hospital at Westmead, Sydney, Australia
- Faculty of Medicine and Health, University of Sydney, Sydney, Australia
| | | | | | | | | |
Collapse
|
50
|
RE-3DLVNet: Refined estimation of the left ventricle volume via interactive 3D segmentation and reinforced quantification. Knowl Based Syst 2022. [DOI: 10.1016/j.knosys.2022.109212] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/18/2022]
|