1
|
Lu X, Liang X, Liu W, Miao X, Guan X. ReeGAN: MRI image edge-preserving synthesis based on GANs trained with misaligned data. Med Biol Eng Comput 2024; 62:1851-1868. [PMID: 38396277 DOI: 10.1007/s11517-024-03035-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/20/2023] [Accepted: 01/27/2024] [Indexed: 02/25/2024]
Abstract
As a crucial medical examination technique, different modalities of magnetic resonance imaging (MRI) complement each other, offering multi-angle and multi-dimensional insights into the body's internal information. Therefore, research on MRI cross-modality conversion is of great significance, and many innovative techniques have been explored. However, most methods are trained on well-aligned data, and the impact of misaligned data has not received sufficient attention. Additionally, many methods focus on transforming the entire image and ignore crucial edge information. To address these challenges, we propose a generative adversarial network based on multi-feature fusion, which effectively preserves edge information while training on noisy data. Notably, we consider images with limited range random transformations as noisy labels and use an additional small auxiliary registration network to help the generator adapt to the noise distribution. Moreover, we inject auxiliary edge information to improve the quality of synthesized target modality images. Our goal is to find the best solution for cross-modality conversion. Comprehensive experiments and ablation studies demonstrate the effectiveness of the proposed method.
Collapse
Affiliation(s)
- Xiangjiang Lu
- Guangxi Key Lab of Multi-Source Information Mining & Security, School of Computer Science and Engineering & School of Software, Guangxi Normal University, Guilin, 541004, China.
| | - Xiaoshuang Liang
- Guangxi Key Lab of Multi-Source Information Mining & Security, School of Computer Science and Engineering & School of Software, Guangxi Normal University, Guilin, 541004, China
| | - Wenjing Liu
- Guangxi Key Lab of Multi-Source Information Mining & Security, School of Computer Science and Engineering & School of Software, Guangxi Normal University, Guilin, 541004, China
| | - Xiuxia Miao
- Guangxi Key Lab of Multi-Source Information Mining & Security, School of Computer Science and Engineering & School of Software, Guangxi Normal University, Guilin, 541004, China
| | - Xianglong Guan
- Guangxi Key Lab of Multi-Source Information Mining & Security, School of Computer Science and Engineering & School of Software, Guangxi Normal University, Guilin, 541004, China
| |
Collapse
|
2
|
Carass A, Greenman D, Dewey BE, Calabresi PA, Prince JL, Pham DL. Image harmonization improves consistency of intra-rater delineations of MS lesions in heterogeneous MRI. NEUROIMAGE. REPORTS 2024; 4:100195. [PMID: 38370461 PMCID: PMC10871705 DOI: 10.1016/j.ynirp.2024.100195] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 02/20/2024]
Abstract
Clinical magnetic resonance images (MRIs) lack a standard intensity scale due to differences in scanner hardware and the pulse sequences used to acquire the images. When MRIs are used for quantification, as in the evaluation of white matter lesions (WMLs) in multiple sclerosis, this lack of intensity standardization becomes a critical problem affecting both the staging and tracking of the disease and its treatment. This paper presents a study of harmonization on WML segmentation consistency, which is evaluated using an object detection classification scheme that incorporates manual delineations from both the original and harmonized MRIs. A cohort of ten people scanned on two different imaging platforms was studied. An expert rater, blinded to the image source, manually delineated WMLs on images from both scanners before and after harmonization. It was found that there is closer agreement in both global and per-lesion WML volume and spatial distribution after harmonization, demonstrating the importance of image harmonization prior to the creation of manual delineations. These results could lead to better truth models in both the development and evaluation of automated lesion segmentation algorithms.
Collapse
Affiliation(s)
- Aaron Carass
- Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, MD 21218, USA
| | - Danielle Greenman
- Center for Neuroscience and Regenerative Medicine, The Henry M. Jackson Foundation for the Advancement of Military Medicine, Bethesda, MD 20817, USA
| | - Blake E. Dewey
- Department of Neurology, Johns Hopkins School of Medicine, Baltimore, MD 21287, USA
| | - Peter A. Calabresi
- Department of Neurology, Johns Hopkins School of Medicine, Baltimore, MD 21287, USA
| | - Jerry L. Prince
- Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, MD 21218, USA
| | - Dzung L. Pham
- Department of Radiology, Uniformed Services University of the Health Sciences, Bethesda, MD 20814, USA
| |
Collapse
|
3
|
Wang C, Tachimori H, Yamaguchi H, Sekiguchi A, Li Y, Yamashita Y. A multimodal deep learning approach for the prediction of cognitive decline and its effectiveness in clinical trials for Alzheimer's disease. Transl Psychiatry 2024; 14:105. [PMID: 38383536 PMCID: PMC10882004 DOI: 10.1038/s41398-024-02819-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 02/24/2023] [Revised: 02/01/2024] [Accepted: 02/07/2024] [Indexed: 02/23/2024] Open
Abstract
Alzheimer's disease is one of the most important health-care challenges in the world. For decades, numerous efforts have been made to develop therapeutics for Alzheimer's disease, but most clinical trials have failed to show significant treatment effects on slowing or halting cognitive decline. Among several challenges in such trials, one recently noticed but unsolved is biased allocation of fast and slow cognitive decliners to treatment and placebo groups during randomization caused by the large individual variation in the speed of cognitive decline. This allocation bias directly results in either over- or underestimation of the treatment effect from the outcome of the trial. In this study, we propose a stratified randomization method using the degree of cognitive decline predicted by an artificial intelligence model as a stratification index to suppress the allocation bias in randomization and evaluate its effectiveness by simulation using ADNI data set.
Collapse
Affiliation(s)
- Caihua Wang
- Bio Science & Engineering Laboratories, FUJIFILM Corporation, Ashigarakami-gun, Kanagawa, Japan
| | - Hisateru Tachimori
- Department of Information Medicine, National Institute of Neuroscience, National Center of Neurology and Psychiatry, Tokyo, Japan
- Endowed Course for Health System Innovation, Keio University School of Medicine, Tokyo, Japan
| | - Hiroyuki Yamaguchi
- Department of Information Medicine, National Institute of Neuroscience, National Center of Neurology and Psychiatry, Tokyo, Japan
- Department of Psychiatry, Yokohama City University School of Medicine, Yokohama, Japan
| | - Atsushi Sekiguchi
- Department of Behavioral Medicine, National Institute of Mental Health, National Center of Neurology and Psychiatry, Tokyo, Japan
| | - Yuanzhong Li
- Bio Science & Engineering Laboratories, FUJIFILM Corporation, Ashigarakami-gun, Kanagawa, Japan.
| | - Yuichi Yamashita
- Department of Information Medicine, National Institute of Neuroscience, National Center of Neurology and Psychiatry, Tokyo, Japan
| |
Collapse
|
4
|
Zuo L, Liu Y, Xue Y, Dewey BE, Remedios SW, Hays SP, Bilgel M, Mowry EM, Newsome SD, Calabresi PA, Resnick SM, Prince JL, Carass A. HACA3: A unified approach for multi-site MR image harmonization. Comput Med Imaging Graph 2023; 109:102285. [PMID: 37657151 PMCID: PMC10592042 DOI: 10.1016/j.compmedimag.2023.102285] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/11/2023] [Revised: 07/11/2023] [Accepted: 08/08/2023] [Indexed: 09/03/2023]
Abstract
The lack of standardization and consistency of acquisition is a prominent issue in magnetic resonance (MR) imaging. This often causes undesired contrast variations in the acquired images due to differences in hardware and acquisition parameters. In recent years, image synthesis-based MR harmonization with disentanglement has been proposed to compensate for the undesired contrast variations. The general idea is to disentangle anatomy and contrast information from MR images to achieve cross-site harmonization. Despite the success of existing methods, we argue that major improvements can be made from three aspects. First, most existing methods are built upon the assumption that multi-contrast MR images of the same subject share the same anatomy. This assumption is questionable, since different MR contrasts are specialized to highlight different anatomical features. Second, these methods often require a fixed set of MR contrasts for training (e.g., both T1-weighted and T2-weighted images), limiting their applicability. Lastly, existing methods are generally sensitive to imaging artifacts. In this paper, we present Harmonization with Attention-based Contrast, Anatomy, and Artifact Awareness (HACA3), a novel approach to address these three issues. HACA3 incorporates an anatomy fusion module that accounts for the inherent anatomical differences between MR contrasts. Furthermore, HACA3 can be trained and applied to any combination of MR contrasts and is robust to imaging artifacts. HACA3 is developed and evaluated on diverse MR datasets acquired from 21 sites with varying field strengths, scanner platforms, and acquisition protocols. Experiments show that HACA3 achieves state-of-the-art harmonization performance under multiple image quality metrics. We also demonstrate the versatility and potential clinical impact of HACA3 on downstream tasks including white matter lesion segmentation for people with multiple sclerosis and longitudinal volumetric analyses for normal aging subjects. Code is available at https://github.com/lianruizuo/haca3.
Collapse
Affiliation(s)
- Lianrui Zuo
- Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, MD 21218, USA; Laboratory of Behavioral Neuroscience, National Institute on Aging, National Institutes of Health, Baltimore, MD 21224, USA.
| | - Yihao Liu
- Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, MD 21218, USA
| | - Yuan Xue
- Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, MD 21218, USA
| | - Blake E Dewey
- Department of Neurology, Johns Hopkins School of Medicine, Baltimore, MD 21287, USA
| | - Samuel W Remedios
- Department of Computer Science, Johns Hopkins University, Baltimore, MD 21218, USA; Radiology and Imaging Sciences, Clinical Center, National Institutes of Health, Bethesda, MD 20892, USA
| | - Savannah P Hays
- Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, MD 21218, USA
| | - Murat Bilgel
- Laboratory of Behavioral Neuroscience, National Institute on Aging, National Institutes of Health, Baltimore, MD 21224, USA
| | - Ellen M Mowry
- Department of Neurology, Johns Hopkins School of Medicine, Baltimore, MD 21287, USA
| | - Scott D Newsome
- Department of Neurology, Johns Hopkins School of Medicine, Baltimore, MD 21287, USA
| | - Peter A Calabresi
- Department of Neurology, Johns Hopkins School of Medicine, Baltimore, MD 21287, USA
| | - Susan M Resnick
- Laboratory of Behavioral Neuroscience, National Institute on Aging, National Institutes of Health, Baltimore, MD 21224, USA
| | - Jerry L Prince
- Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, MD 21218, USA
| | - Aaron Carass
- Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, MD 21218, USA
| |
Collapse
|
5
|
Liu J, Pasumarthi S, Duffy B, Gong E, Datta K, Zaharchuk G. One Model to Synthesize Them All: Multi-Contrast Multi-Scale Transformer for Missing Data Imputation. IEEE TRANSACTIONS ON MEDICAL IMAGING 2023; 42:2577-2591. [PMID: 37030684 PMCID: PMC10543020 DOI: 10.1109/tmi.2023.3261707] [Citation(s) in RCA: 10] [Impact Index Per Article: 10.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/19/2023]
Abstract
Multi-contrast magnetic resonance imaging (MRI) is widely used in clinical practice as each contrast provides complementary information. However, the availability of each imaging contrast may vary amongst patients, which poses challenges to radiologists and automated image analysis algorithms. A general approach for tackling this problem is missing data imputation, which aims to synthesize the missing contrasts from existing ones. While several convolutional neural networks (CNN) based algorithms have been proposed, they suffer from the fundamental limitations of CNN models, such as the requirement for fixed numbers of input and output channels, the inability to capture long-range dependencies, and the lack of interpretability. In this work, we formulate missing data imputation as a sequence-to-sequence learning problem and propose a multi-contrast multi-scale Transformer (MMT), which can take any subset of input contrasts and synthesize those that are missing. MMT consists of a multi-scale Transformer encoder that builds hierarchical representations of inputs combined with a multi-scale Transformer decoder that generates the outputs in a coarse-to-fine fashion. The proposed multi-contrast Swin Transformer blocks can efficiently capture intra- and inter-contrast dependencies for accurate image synthesis. Moreover, MMT is inherently interpretable as it allows us to understand the importance of each input contrast in different regions by analyzing the in-built attention maps of Transformer blocks in the decoder. Extensive experiments on two large-scale multi-contrast MRI datasets demonstrate that MMT outperforms the state-of-the-art methods quantitatively and qualitatively.
Collapse
|
6
|
Kawahara D, Yoshimura H, Matsuura T, Saito A, Nagata Y. MRI image synthesis for fluid-attenuated inversion recovery and diffusion-weighted images with deep learning. Phys Eng Sci Med 2023; 46:313-323. [PMID: 36715853 DOI: 10.1007/s13246-023-01220-z] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/19/2020] [Accepted: 01/10/2023] [Indexed: 01/31/2023]
Abstract
This study aims to synthesize fluid-attenuated inversion recovery (FLAIR) and diffusion-weighted images (DWI) with a deep conditional adversarial network from T1- and T2-weighted magnetic resonance imaging (MRI) images. A total of 1980 images of 102 patients were split into two datasets: 1470 (68 patients) in a training set and 510 (34 patients) in a test set. The prediction framework was based on a convolutional neural network with a generator and discriminator. T1-weighted, T2-weighted, and composite images were used as inputs. The digital imaging and communications in medicine (DICOM) images were converted to 8-bit red-green-blue images. The red and blue channels of the composite images were assigned to 8-bit grayscale pixel values in T1-weighted images, and the green channel was assigned to those in T2-weighted images. The prediction FLAIR and DWI images were of the same objects as the inputs. For the results, the prediction model with composite MRI input images in the DWI image showed the smallest relative mean absolute error (rMAE) and largest mutual information (MI), and that in the FLAIR image showed the largest relative mean-square error (rMSE), relative root-mean-square error (rRMSE), and peak signal-to-noise ratio (PSNR). For the FLAIR image, the prediction model with the T2-weighted MRI input images generated more accurate synthesis results than that with the T1-weighted inputs. The proposed image synthesis framework can improve the versatility and quality of multi-contrast MRI without extra scans. The composite input MRI image contributes to synthesizing the multi-contrast MRI image efficiently.
Collapse
Affiliation(s)
- Daisuke Kawahara
- Department of Radiation Oncology, Graduate School of Biomedical and Health Sciences, Hiroshima University, Hiroshima, 734-8551, Japan.
| | - Hisanori Yoshimura
- Department of Radiation Oncology, Graduate School of Biomedical and Health Sciences, Hiroshima University, Hiroshima, 734-8551, Japan
- Department of Radiology, National Hospital Organization Kure Medical Center, Hiroshima, 737-0023, Japan
| | - Takaaki Matsuura
- Department of Radiation Oncology, Graduate School of Biomedical and Health Sciences, Hiroshima University, Hiroshima, 734-8551, Japan
| | - Akito Saito
- Department of Radiation Oncology, Graduate School of Biomedical and Health Sciences, Hiroshima University, Hiroshima, 734-8551, Japan
| | - Yasushi Nagata
- Department of Radiation Oncology, Graduate School of Biomedical and Health Sciences, Hiroshima University, Hiroshima, 734-8551, Japan
- Hiroshima High-Precision Radiotherapy Cancer Center, Hiroshima, 732-0057, Japan
| |
Collapse
|
7
|
Yurt M, Dalmaz O, Dar S, Ozbey M, Tinaz B, Oguz K, Cukur T. Semi-Supervised Learning of MRI Synthesis Without Fully-Sampled Ground Truths. IEEE TRANSACTIONS ON MEDICAL IMAGING 2022; 41:3895-3906. [PMID: 35969576 DOI: 10.1109/tmi.2022.3199155] [Citation(s) in RCA: 8] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/15/2023]
Abstract
Learning-based translation between MRI contrasts involves supervised deep models trained using high-quality source- and target-contrast images derived from fully-sampled acquisitions, which might be difficult to collect under limitations on scan costs or time. To facilitate curation of training sets, here we introduce the first semi-supervised model for MRI contrast translation (ssGAN) that can be trained directly using undersampled k-space data. To enable semi-supervised learning on undersampled data, ssGAN introduces novel multi-coil losses in image, k-space, and adversarial domains. The multi-coil losses are selectively enforced on acquired k-space samples unlike traditional losses in single-coil synthesis models. Comprehensive experiments on retrospectively undersampled multi-contrast brain MRI datasets are provided. Our results demonstrate that ssGAN yields on par performance to a supervised model, while outperforming single-coil models trained on coil-combined magnitude images. It also outperforms cascaded reconstruction-synthesis models where a supervised synthesis model is trained following self-supervised reconstruction of undersampled data. Thus, ssGAN holds great promise to improve the feasibility of learning-based multi-contrast MRI synthesis.
Collapse
|
8
|
Zhang X, He X, Guo J, Ettehadi N, Aw N, Semanek D, Posner J, Laine A, Wang Y. PTNet3D: A 3D High-Resolution Longitudinal Infant Brain MRI Synthesizer Based on Transformers. IEEE TRANSACTIONS ON MEDICAL IMAGING 2022; 41:2925-2940. [PMID: 35560070 PMCID: PMC9529847 DOI: 10.1109/tmi.2022.3174827] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/15/2023]
Abstract
An increased interest in longitudinal neurodevelopment during the first few years after birth has emerged in recent years. Noninvasive magnetic resonance imaging (MRI) can provide crucial information about the development of brain structures in the early months of life. Despite the success of MRI collections and analysis for adults, it remains a challenge for researchers to collect high-quality multimodal MRIs from developing infant brains because of their irregular sleep pattern, limited attention, inability to follow instructions to stay still during scanning. In addition, there are limited analytic approaches available. These challenges often lead to a significant reduction of usable MRI scans and pose a problem for modeling neurodevelopmental trajectories. Researchers have explored solving this problem by synthesizing realistic MRIs to replace corrupted ones. Among synthesis methods, the convolutional neural network-based (CNN-based) generative adversarial networks (GANs) have demonstrated promising performance. In this study, we introduced a novel 3D MRI synthesis framework- pyramid transformer network (PTNet3D)- which relies on attention mechanisms through transformer and performer layers. We conducted extensive experiments on high-resolution Developing Human Connectome Project (dHCP) and longitudinal Baby Connectome Project (BCP) datasets. Compared with CNN-based GANs, PTNet3D consistently shows superior synthesis accuracy and superior generalization on two independent, large-scale infant brain MRI datasets. Notably, we demonstrate that PTNet3D synthesized more realistic scans than CNN-based models when the input is from multi-age subjects. Potential applications of PTNet3D include synthesizing corrupted or missing images. By replacing corrupted scans with synthesized ones, we observed significant improvement in infant whole brain segmentation.
Collapse
|
9
|
Bi-MGAN: Bidirectional T1-to-T2 MRI images prediction using multi-generative multi-adversarial nets. Biomed Signal Process Control 2022. [DOI: 10.1016/j.bspc.2022.103994] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/18/2022]
|
10
|
Wang C, Li Y, Tsuboshita Y, Sakurai T, Goto T, Yamaguchi H, Yamashita Y, Sekiguchi A, Tachimori H. A high-generalizability machine learning framework for predicting the progression of Alzheimer's disease using limited data. NPJ Digit Med 2022; 5:43. [PMID: 35414651 PMCID: PMC9005545 DOI: 10.1038/s41746-022-00577-x] [Citation(s) in RCA: 9] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/17/2021] [Accepted: 02/11/2022] [Indexed: 11/25/2022] Open
Abstract
Alzheimer’s disease is a neurodegenerative disease that imposes a substantial financial burden on society. A number of machine learning studies have been conducted to predict the speed of its progression, which varies widely among different individuals, for recruiting fast progressors in future clinical trials. However, because the data in this field are very limited, two problems have yet to be solved: the first is that models built on limited data tend to induce overfitting and have low generalizability, and the second is that no cross-cohort evaluations have been done. Here, to suppress the overfitting caused by limited data, we propose a hybrid machine learning framework consisting of multiple convolutional neural networks that automatically extract image features from the point of view of brain segments, which are relevant to cognitive decline according to clinical findings, and a linear support vector classifier that uses extracted image features together with non-image information to make robust final predictions. The experimental results indicate that our model achieves superior performance (accuracy: 0.88, area under the curve [AUC]: 0.95) compared with other state-of-the-art methods. Moreover, our framework demonstrates high generalizability as a result of evaluations using a completely different cohort dataset (accuracy: 0.84, AUC: 0.91) collected from a different population than that used for training.
Collapse
Affiliation(s)
- Caihua Wang
- Imaging Technology Center, FUJIFILM Corporation, Kanagawa, Japan.
| | - Yuanzhong Li
- Imaging Technology Center, FUJIFILM Corporation, Kanagawa, Japan.
| | | | - Takuya Sakurai
- Imaging Technology Center, FUJIFILM Corporation, Kanagawa, Japan
| | - Tsubasa Goto
- Imaging Technology Center, FUJIFILM Corporation, Kanagawa, Japan
| | - Hiroyuki Yamaguchi
- Department of Information Medicine, National Institute of Neuroscience, National Center of Neurology and Psychiatry, Tokyo, Japan.,Department of Psychiatry, Yokohama City University School of Medicine, Yokohama, Japan
| | - Yuichi Yamashita
- Department of Information Medicine, National Institute of Neuroscience, National Center of Neurology and Psychiatry, Tokyo, Japan
| | - Atsushi Sekiguchi
- Department of Behavioral Medicine, National Institute of Mental Health, National Center of Neurology and Psychiatry, Tokyo, Japan
| | - Hisateru Tachimori
- Department of Clinical Epidemiology, Translational Medical Center, National Center of Neurology and Psychiatry, Tokyo, Japan.,Endowed Course for Health System Innovation, Keio University School of Medicine, Tokyo, Japan
| | | |
Collapse
|
11
|
Yurt M, Özbey M, UH Dar S, Tinaz B, Oguz KK, Çukur T. Progressively Volumetrized Deep Generative Models for Data-Efficient Contextual Learning of MR Image Recovery. Med Image Anal 2022; 78:102429. [DOI: 10.1016/j.media.2022.102429] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/10/2021] [Revised: 03/14/2022] [Accepted: 03/18/2022] [Indexed: 10/18/2022]
|
12
|
Wu G, Chen X, Shi Z, Zhang D, Hu Z, Mao Y, Wang Y, Yu J. Convolutional neural network with coarse-to-fine resolution fusion and residual learning structures for cross-modality image synthesis. Biomed Signal Process Control 2022. [DOI: 10.1016/j.bspc.2021.103199] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/02/2022]
|
13
|
Manjón JV, Romero JE, Coupe P. Deep learning based MRI contrast synthesis using full volume prediction using full volume prediction. Biomed Phys Eng Express 2021; 8. [PMID: 34814130 DOI: 10.1088/2057-1976/ac3c64] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/13/2021] [Accepted: 11/23/2021] [Indexed: 11/12/2022]
Abstract
In Magnetic Resonance Imaging (MRI), depending on the image acquisition settings, a large number of image types or contrasts can be generated showing complementary information of the same imaged subject. This multi-spectral information is highly beneficial since can improve MRI analysis tasks such as segmentation and registration, thanks to pattern ambiguity reduction. However, the acquisition of several contrasts is not always possible due to time limitations and patient comfort constraints. Contrast synthesis has emerged recently as an approximate solution to generate other image types different from those acquired originally. Most of the previously proposed methods for contrast synthesis are slice-based which result in intensity inconsistencies between neighbor slices when applied in 3D. We propose the use of a 3D convolutional neural network (CNN) capable of generating T2 and FLAIR images from a single anatomical T1 source volume. The proposed network is a 3D variant of the UNet that processes the whole volume at once breaking with the inconsistency in the resulting output volumes related to 2D slice or patch-based methods. Since working with a full volume at once has a huge memory demand we have introduced a spatial-to-depth and a reconstruction layer that allows working with the full volume but maintain the required network complexity to solve the problem. Our approach enhances the coherence in the synthesized volume while improving the accuracy thanks to the integrated three-dimensional context-awareness. Finally, the proposed method has been validated with a segmentation method, thus demonstrating its usefulness in a direct and relevant application.
Collapse
Affiliation(s)
- José V Manjón
- Instituto de Aplicaciones de las Tecnologías de la Información y de las Comunicaciones Avanzadas (ITACA), Universitat Politécnica de Valencia, Camino de Vera s/n, 46022, Spain
| | - José E Romero
- Instituto de Aplicaciones de las Tecnologías de la Información y de las Comunicaciones Avanzadas (ITACA), Universitat Politécnica de Valencia, Camino de Vera s/n, 46022, Spain
| | - Pierrick Coupe
- CNRS, Univ. Bordeaux, Bordeaux INP, LaBRI, UMR5800, PICTURA Research Group, 351, cours de la Liberation F-33405 Talence cedex, France
| |
Collapse
|
14
|
Zuo L, Dewey BE, Liu Y, He Y, Newsome SD, Mowry EM, Resnick SM, Prince JL, Carass A. Unsupervised MR harmonization by learning disentangled representations using information bottleneck theory. Neuroimage 2021; 243:118569. [PMID: 34506916 PMCID: PMC10473284 DOI: 10.1016/j.neuroimage.2021.118569] [Citation(s) in RCA: 32] [Impact Index Per Article: 10.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/06/2021] [Revised: 08/11/2021] [Accepted: 09/07/2021] [Indexed: 01/19/2023] Open
Abstract
In magnetic resonance (MR) imaging, a lack of standardization in acquisition often causes pulse sequence-based contrast variations in MR images from site to site, which impedes consistent measurements in automatic analyses. In this paper, we propose an unsupervised MR image harmonization approach, CALAMITI (Contrast Anatomy Learning and Analysis for MR Intensity Translation and Integration), which aims to alleviate contrast variations in multi-site MR imaging. Designed using information bottleneck theory, CALAMITI learns a globally disentangled latent space containing both anatomical and contrast information, which permits harmonization. In contrast to supervised harmonization methods, our approach does not need a sample population to be imaged across sites. Unlike traditional unsupervised harmonization approaches which often suffer from geometry shifts, CALAMITI better preserves anatomy by design. The proposed method is also able to adapt to a new testing site with a straightforward fine-tuning process. Experiments on MR images acquired from ten sites show that CALAMITI achieves superior performance compared with other harmonization approaches.
Collapse
Affiliation(s)
- Lianrui Zuo
- Department of Electrical and Computer Engineering, The Johns Hopkins University, Baltimore, MD 21218 USA; Laboratory of Behavioral Neuroscience, National Institute on Aging, National Institute of Health, Baltimore, MD 20892, USA.
| | - Blake E Dewey
- Department of Electrical and Computer Engineering, The Johns Hopkins University, Baltimore, MD 21218 USA
| | - Yihao Liu
- Department of Electrical and Computer Engineering, The Johns Hopkins University, Baltimore, MD 21218 USA
| | - Yufan He
- Department of Electrical and Computer Engineering, The Johns Hopkins University, Baltimore, MD 21218 USA
| | - Scott D Newsome
- Department of Neurology, The Johns Hopkins School of Medicine, Baltimore, MD 21287, USA
| | - Ellen M Mowry
- Department of Neurology, The Johns Hopkins School of Medicine, Baltimore, MD 21287, USA
| | - Susan M Resnick
- Laboratory of Behavioral Neuroscience, National Institute on Aging, National Institute of Health, Baltimore, MD 20892, USA
| | - Jerry L Prince
- Department of Electrical and Computer Engineering, The Johns Hopkins University, Baltimore, MD 21218 USA
| | - Aaron Carass
- Department of Electrical and Computer Engineering, The Johns Hopkins University, Baltimore, MD 21218 USA
| |
Collapse
|
15
|
Guo P, Wang P, Yasarla R, Zhou J, Patel VM, Jiang S. Anatomic and Molecular MR Image Synthesis Using Confidence Guided CNNs. IEEE TRANSACTIONS ON MEDICAL IMAGING 2021; 40:2832-2844. [PMID: 33351754 PMCID: PMC8543492 DOI: 10.1109/tmi.2020.3046460] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/12/2023]
Abstract
Data-driven automatic approaches have demonstrated their great potential in resolving various clinical diagnostic dilemmas in neuro-oncology, especially with the help of standard anatomic and advanced molecular MR images. However, data quantity and quality remain a key determinant, and a significant limit of the potential applications. In our previous work, we explored the synthesis of anatomic and molecular MR image networks (SAMR) in patients with post-treatment malignant gliomas. In this work, we extend this through a confidence-guided SAMR (CG-SAMR) that synthesizes data from lesion contour information to multi-modal MR images, including T1-weighted ( [Formula: see text]), gadolinium enhanced [Formula: see text] (Gd- [Formula: see text]), T2-weighted ( [Formula: see text]), and fluid-attenuated inversion recovery ( FLAIR ), as well as the molecular amide proton transfer-weighted ( [Formula: see text]) sequence. We introduce a module that guides the synthesis based on a confidence measure of the intermediate results. Furthermore, we extend the proposed architecture to allow training using unpaired data. Extensive experiments on real clinical data demonstrate that the proposed model can perform better than current the state-of-the-art synthesis methods. Our code is available at https://github.com/guopengf/CG-SAMR.
Collapse
|
16
|
Kim S, Jang H, Hong S, Hong YS, Bae WC, Kim S, Hwang D. Fat-saturated image generation from multi-contrast MRIs using generative adversarial networks with Bloch equation-based autoencoder regularization. Med Image Anal 2021; 73:102198. [PMID: 34403931 DOI: 10.1016/j.media.2021.102198] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/15/2020] [Revised: 07/18/2021] [Accepted: 07/23/2021] [Indexed: 11/28/2022]
Abstract
Obtaining multiple series of magnetic resonance (MR) images with different contrasts is useful for accurate diagnosis of human spinal conditions. However, this can be time consuming and a burden on both the patient and the hospital. We propose a Bloch equation-based autoencoder regularization generative adversarial network (BlochGAN) to generate a fat saturation T2-weighted (T2 FS) image from T1-weighted (T1-w) and T2-weighted (T2-w) images of human spine. To achieve this, our approach was to utilize the relationship between the contrasts using Bloch equation since it is a fundamental principle of MR physics and serves as a physical basis of each contrasts. BlochGAN properly generated the target-contrast images using the autoencoder regularization based on the Bloch equation to identify the physical basis of the contrasts. BlochGAN consists of four sub-networks: an encoder, a decoder, a generator, and a discriminator. The encoder extracts features from the multi-contrast input images, and the generator creates target T2 FS images using the features extracted from the encoder. The discriminator assists network learning by providing adversarial loss, and the decoder reconstructs the input multi-contrast images and regularizes the learning process by providing reconstruction loss. The discriminator and the decoder are only used in the training process. Our results demonstrate that BlochGAN achieved quantitatively and qualitatively superior performance compared to conventional medical image synthesis methods in generating spine T2 FS images from T1-w, and T2-w images.
Collapse
Affiliation(s)
- Sewon Kim
- School of Electrical and Electronic Engineering, Yonsei University, 50, Yonsei-ro, Seodaemun-gu, Seoul 03722, Republic of Korea
| | - Hanbyol Jang
- School of Electrical and Electronic Engineering, Yonsei University, 50, Yonsei-ro, Seodaemun-gu, Seoul 03722, Republic of Korea
| | - Seokjun Hong
- School of Electrical and Electronic Engineering, Yonsei University, 50, Yonsei-ro, Seodaemun-gu, Seoul 03722, Republic of Korea
| | - Yeong Sang Hong
- Center for Clinical Imaging Data Science Center, Research Institute of Radiological Science, Department of Radiology, Yonsei University College of Medicine, 50-1, Yonsei-ro, Seodaemun-gu, Seoul 03722, Republic of Korea; Department of Radiology, Gangnam Severance Hospital, 211, Eonju-ro, Gangnam-gu, Seoul 06273, Republic of Korea
| | - Won C Bae
- Department of Radiology, Veterans Affairs San Diego Healthcare System, 3350 La Jolla Village Drive, San Diego, CA 92161-0114, USA; Department of Radiology, University of California-San Diego, La Jolla, CA 92093-0997, USA
| | - Sungjun Kim
- Center for Clinical Imaging Data Science Center, Research Institute of Radiological Science, Department of Radiology, Yonsei University College of Medicine, 50-1, Yonsei-ro, Seodaemun-gu, Seoul 03722, Republic of Korea; Department of Radiology, Gangnam Severance Hospital, 211, Eonju-ro, Gangnam-gu, Seoul 06273, Republic of Korea.
| | - Dosik Hwang
- School of Electrical and Electronic Engineering, Yonsei University, 50, Yonsei-ro, Seodaemun-gu, Seoul 03722, Republic of Korea; Center for Clinical Imaging Data Science Center, Research Institute of Radiological Science, Department of Radiology, Yonsei University College of Medicine, 50-1, Yonsei-ro, Seodaemun-gu, Seoul 03722, Republic of Korea.
| |
Collapse
|
17
|
Bernal J, Valverde S, Kushibar K, Cabezas M, Oliver A, Lladó X. Generating Longitudinal Atrophy Evaluation Datasets on Brain Magnetic Resonance Images Using Convolutional Neural Networks and Segmentation Priors. Neuroinformatics 2021; 19:477-492. [PMID: 33389607 DOI: 10.1007/s12021-020-09499-z] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 11/06/2020] [Indexed: 02/03/2023]
Abstract
Brain atrophy quantification plays a fundamental role in neuroinformatics since it permits studying brain development and neurological disorders. However, the lack of a ground truth prevents testing the accuracy of longitudinal atrophy quantification methods. We propose a deep learning framework to generate longitudinal datasets by deforming T1-w brain magnetic resonance imaging scans as requested through segmentation maps. Our proposal incorporates a cascaded multi-path U-Net optimised with a multi-objective loss which allows its paths to generate different brain regions accurately. We provided our model with baseline scans and real follow-up segmentation maps from two longitudinal datasets, ADNI and OASIS, and observed that our framework could produce synthetic follow-up scans that matched the real ones (Total scans= 584; Median absolute error: 0.03 ± 0.02; Structural similarity index: 0.98 ± 0.02; Dice similarity coefficient: 0.95 ± 0.02; Percentage of brain volume change: 0.24 ± 0.16; Jacobian integration: 1.13 ± 0.05). Compared to two relevant works generating brain lesions using U-Nets and conditional generative adversarial networks (CGAN), our proposal outperformed them significantly in most cases (p < 0.01), except in the delineation of brain edges where the CGAN took the lead (Jacobian integration: Ours - 1.13 ± 0.05 vs CGAN - 1.00 ± 0.02; p < 0.01). We examined whether changes induced with our framework were detected by FAST, SPM, SIENA, SIENAX, and the Jacobian integration method. We observed that induced and detected changes were highly correlated (Adj. R2 > 0.86). Our preliminary results on harmonised datasets showed the potential of our framework to be applied to various data collections without further adjustment.
Collapse
Affiliation(s)
- Jose Bernal
- Computer Vision and Robotics Institute, Universitat de Girona, Girona, Spain.
| | - Sergi Valverde
- Computer Vision and Robotics Institute, Universitat de Girona, Girona, Spain
| | - Kaisar Kushibar
- Computer Vision and Robotics Institute, Universitat de Girona, Girona, Spain
| | - Mariano Cabezas
- Computer Vision and Robotics Institute, Universitat de Girona, Girona, Spain
| | - Arnau Oliver
- Computer Vision and Robotics Institute, Universitat de Girona, Girona, Spain
| | - Xavier Lladó
- Computer Vision and Robotics Institute, Universitat de Girona, Girona, Spain
| | | |
Collapse
|
18
|
Roy A, Reich BJ, Guinness J, Shinohara RT, Staicu AM. Spatial Shrinkage Via the Product Independent Gaussian Process Prior. J Comput Graph Stat 2021. [DOI: 10.1080/10618600.2021.1923512] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/21/2022]
Affiliation(s)
- Arkaprava Roy
- Department of Biostatistics, University of Florida, Gainesville, FL
| | - Brian J. Reich
- Department of Statistics, North Carolina State University, Raleigh, NC
| | - Joseph Guinness
- Department of Statistics and Data Science, Cornell University, Ithaca, NY
| | | | - Ana-Maria Staicu
- Department of Statistics, North Carolina State University, Raleigh, NC
| |
Collapse
|
19
|
Fei Y, Zhan B, Hong M, Wu X, Zhou J, Wang Y. Deep learning-based multi-modal computing with feature disentanglement for MRI image synthesis. Med Phys 2021; 48:3778-3789. [PMID: 33959965 DOI: 10.1002/mp.14929] [Citation(s) in RCA: 9] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/06/2020] [Revised: 04/06/2021] [Accepted: 04/18/2021] [Indexed: 12/12/2022] Open
Abstract
PURPOSE Different Magnetic resonance imaging (MRI) modalities of the same anatomical structure are required to present different pathological information from the physical level for diagnostic needs. However, it is often difficult to obtain full-sequence MRI images of patients owing to limitations such as time consumption and high cost. The purpose of this work is to develop an algorithm for target MRI sequences prediction with high accuracy, and provide more information for clinical diagnosis. METHODS We propose a deep learning-based multi-modal computing model for MRI synthesis with feature disentanglement strategy. To take full advantage of the complementary information provided by different modalities, multi-modal MRI sequences are utilized as input. Notably, the proposed approach decomposes each input modality into modality-invariant space with shared information and modality-specific space with specific information, so that features are extracted separately to effectively process the input data. Subsequently, both of them are fused through the adaptive instance normalization (AdaIN) layer in the decoder. In addition, to address the lack of specific information of the target modality in the test phase, a local adaptive fusion (LAF) module is adopted to generate a modality-like pseudo-target with specific information similar to the ground truth. RESULTS To evaluate the synthesis performance, we verify our method on the BRATS2015 dataset of 164 subjects. The experimental results demonstrate our approach significantly outperforms the benchmark method and other state-of-the-art medical image synthesis methods in both quantitative and qualitative measures. Compared with the pix2pixGANs method, the PSNR improves from 23.68 to 24.8. Moreover the ablation studies have also verified the effectiveness of important components of the proposed method. CONCLUSION The proposed method could be effective in prediction of target MRI sequences, and useful for clinical diagnosis and treatment.
Collapse
Affiliation(s)
- Yuchen Fei
- School of Computer Science, Sichuan University, Chengdu, Sichuan, 610065, China
| | - Bo Zhan
- School of Computer Science, Sichuan University, Chengdu, Sichuan, 610065, China
| | - Mei Hong
- School of Computer Science, Sichuan University, Chengdu, Sichuan, 610065, China
| | - Xi Wu
- School of Computer Science, Chengdu University of Information Technology, Chengdu, Sichuan, China
| | - Jiliu Zhou
- School of Computer Science, Sichuan University, Chengdu, Sichuan, 610065, China.,School of Computer Science, Chengdu University of Information Technology, Chengdu, Sichuan, China
| | - Yan Wang
- School of Computer Science, Sichuan University, Chengdu, Sichuan, 610065, China
| |
Collapse
|
20
|
Yurt M, Dar SU, Erdem A, Erdem E, Oguz KK, Çukur T. mustGAN: multi-stream Generative Adversarial Networks for MR Image Synthesis. Med Image Anal 2021; 70:101944. [PMID: 33690024 DOI: 10.1016/j.media.2020.101944] [Citation(s) in RCA: 46] [Impact Index Per Article: 15.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/16/2020] [Revised: 12/11/2020] [Accepted: 12/15/2020] [Indexed: 01/28/2023]
Abstract
Multi-contrast MRI protocols increase the level of morphological information available for diagnosis. Yet, the number and quality of contrasts are limited in practice by various factors including scan time and patient motion. Synthesis of missing or corrupted contrasts from other high-quality ones can alleviate this limitation. When a single target contrast is of interest, common approaches for multi-contrast MRI involve either one-to-one or many-to-one synthesis methods depending on their input. One-to-one methods take as input a single source contrast, and they learn a latent representation sensitive to unique features of the source. Meanwhile, many-to-one methods receive multiple distinct sources, and they learn a shared latent representation more sensitive to common features across sources. For enhanced image synthesis, we propose a multi-stream approach that aggregates information across multiple source images via a mixture of multiple one-to-one streams and a joint many-to-one stream. The complementary feature maps generated in the one-to-one streams and the shared feature maps generated in the many-to-one stream are combined with a fusion block. The location of the fusion block is adaptively modified to maximize task-specific performance. Quantitative and radiological assessments on T1,- T2-, PD-weighted, and FLAIR images clearly demonstrate the superior performance of the proposed method compared to previous state-of-the-art one-to-one and many-to-one methods.
Collapse
Affiliation(s)
- Mahmut Yurt
- Department of Electrical and Electronics Engineering, Bilkent University, Ankara, TR-06800, Turkey; National Magnetic Resonance Research Center, Bilkent University, Ankara, TR-06800, Turkey
| | - Salman Uh Dar
- Department of Electrical and Electronics Engineering, Bilkent University, Ankara, TR-06800, Turkey; National Magnetic Resonance Research Center, Bilkent University, Ankara, TR-06800, Turkey
| | - Aykut Erdem
- Department of Computer Engineering, Koç University, İstanbul, TR-34450, Turkey
| | - Erkut Erdem
- Department of Computer Engineering, Hacettepe University, Ankara, TR-06800, Turkey
| | - Kader K Oguz
- National Magnetic Resonance Research Center, Bilkent University, Ankara, TR-06800, Turkey; Department of Radiology, Hacettepe University, Ankara, TR-06100, Turkey
| | - Tolga Çukur
- Department of Electrical and Electronics Engineering, Bilkent University, Ankara, TR-06800, Turkey; National Magnetic Resonance Research Center, Bilkent University, Ankara, TR-06800, Turkey; Neuroscience Program, Aysel Sabuncu Brain Research Center, Bilkent, Ankara, TR-06800, Turkey.
| |
Collapse
|
21
|
Dai X, Lei Y, Fu Y, Curran WJ, Liu T, Mao H, Yang X. Multimodal MRI synthesis using unified generative adversarial networks. Med Phys 2020; 47:6343-6354. [PMID: 33053202 PMCID: PMC7796974 DOI: 10.1002/mp.14539] [Citation(s) in RCA: 28] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/13/2020] [Revised: 08/27/2020] [Accepted: 10/01/2020] [Indexed: 11/11/2022] Open
Abstract
PURPOSE Complementary information obtained from multiple contrasts of tissue facilitates physicians assessing, diagnosing and planning treatment of a variety of diseases. However, acquiring multiple contrasts magnetic resonance images (MRI) for every patient using multiple pulse sequences is time-consuming and expensive, where, medical image synthesis has been demonstrated as an effective alternative. The purpose of this study is to develop a unified framework for multimodal MR image synthesis. METHODS A unified generative adversarial network consisting of only a single generator and a single discriminator was developed to learn the mappings among images of four different modalities. The generator took an image and its modality label as inputs and learned to synthesize the image in the target modality, while the discriminator was trained to distinguish between real and synthesized images and classify them to their corresponding modalities. The network was trained and tested using multimodal brain MRI consisting of four different contrasts which are T1-weighted (T1), T1-weighted and contrast-enhanced (T1c), T2-weighted (T2), and fluid-attenuated inversion recovery (Flair). Quantitative assessments of our proposed method were made through computing normalized mean absolute error (NMAE), peak signal-to-noise ratio (PSNR), structural similarity index measurement (SSIM), visual information fidelity (VIF), and naturalness image quality evaluator (NIQE). RESULTS The proposed model was trained and tested on a cohort of 274 glioma patients with well-aligned multi-types of MRI scans. After the model was trained, tests were conducted by using each of T1, T1c, T2, Flair as a single input modality to generate its respective rest modalities. Our proposed method shows high accuracy and robustness for image synthesis with arbitrary MRI modality that is available in the database as input. For example, with T1 as input modality, the NMAEs for the generated T1c, T2, Flair respectively are 0.034 ± 0.005, 0.041 ± 0.006, and 0.041 ± 0.006, the PSNRs respectively are 32.353 ± 2.525 dB, 30.016 ± 2.577 dB, and 29.091 ± 2.795 dB, the SSIMs are 0.974 ± 0.059, 0.969 ± 0.059, and 0.959 ± 0.059, the VIF are 0.750 ± 0.087, 0.706 ± 0.097, and 0.654 ± 0.062, and NIQE are 1.396 ± 0.401, 1.511 ± 0.460, and 1.259 ± 0.358, respectively. CONCLUSIONS We proposed a novel multimodal MR image synthesis method based on a unified generative adversarial network. The network takes an image and its modality label as inputs and synthesizes multimodal images in a single forward pass. The results demonstrate that the proposed method is able to accurately synthesize multimodal MR images from a single MR image.
Collapse
Affiliation(s)
- Xianjin Dai
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, GA
| | - Yang Lei
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, GA
| | - Yabo Fu
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, GA
| | - Walter J. Curran
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, GA
| | - Tian Liu
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, GA
| | - Hui Mao
- Department of Radiology and Imaging Sciences and Winship Cancer Institute, Emory University, Atlanta, GA 30322
| | - Xiaofeng Yang
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, GA
| |
Collapse
|
22
|
Zhou T, Fu H, Chen G, Shen J, Shao L. Hi-Net: Hybrid-Fusion Network for Multi-Modal MR Image Synthesis. IEEE TRANSACTIONS ON MEDICAL IMAGING 2020; 39:2772-2781. [PMID: 32086202 DOI: 10.1109/tmi.2020.2975344] [Citation(s) in RCA: 96] [Impact Index Per Article: 24.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/12/2023]
Abstract
Magnetic resonance imaging (MRI) is a widely used neuroimaging technique that can provide images of different contrasts (i.e., modalities). Fusing this multi-modal data has proven particularly effective for boosting model performance in many tasks. However, due to poor data quality and frequent patient dropout, collecting all modalities for every patient remains a challenge. Medical image synthesis has been proposed as an effective solution, where any missing modalities are synthesized from the existing ones. In this paper, we propose a novel Hybrid-fusion Network (Hi-Net) for multi-modal MR image synthesis, which learns a mapping from multi-modal source images (i.e., existing modalities) to target images (i.e., missing modalities). In our Hi-Net, a modality-specific network is utilized to learn representations for each individual modality, and a fusion network is employed to learn the common latent representation of multi-modal data. Then, a multi-modal synthesis network is designed to densely combine the latent representation with hierarchical features from each modality, acting as a generator to synthesize the target images. Moreover, a layer-wise multi-modal fusion strategy effectively exploits the correlations among multiple modalities, where a Mixed Fusion Block (MFB) is proposed to adaptively weight different fusion strategies. Extensive experiments demonstrate the proposed model outperforms other state-of-the-art medical image synthesis methods.
Collapse
|
23
|
Neubert A, Bourgeat P, Wood J, Engstrom C, Chandra SS, Crozier S, Fripp J. Simultaneous super-resolution and contrast synthesis of routine clinical magnetic resonance images of the knee for improving automatic segmentation of joint cartilage: data from the Osteoarthritis Initiative. Med Phys 2020; 47:4939-4948. [PMID: 32745260 DOI: 10.1002/mp.14421] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/19/2020] [Revised: 07/07/2020] [Accepted: 07/24/2020] [Indexed: 12/21/2022] Open
Abstract
PURPOSE High resolution three-dimensional (3D) magnetic resonance (MR) images are well suited for automated cartilage segmentation in the human knee joint. However, volumetric scans such as 3D Double-Echo Steady-State (DESS) images are not routinely acquired in clinical practice which limits opportunities for reliable cartilage segmentation using (fully) automated algorithms. In this work, a method for generating synthetic 3D MR (syn3D-DESS) images with better contrast and higher spatial resolution from routine, low resolution, two-dimensional (2D) Turbo-Spin Echo (TSE) clinical knee scans is proposed. METHODS A UNet convolutional neural network is employed for synthesizing enhanced artificial MR images suitable for automated knee cartilage segmentation. Training of the model was performed on a large, publically available dataset from the OAI, consisting of 578 MR examinations of knee joints from 102 healthy individuals and patients with knee osteoarthritis. RESULTS The generated synthetic images have higher spatial resolution and better tissue contrast than the original 2D TSE, which allow high quality automated 3D segmentations of the cartilage. The proposed approach was evaluated on a separate set of MR images from 88 subjects with manual cartilage segmentations. It provided a significant improvement in automated segmentation of knee cartilages when using the syn3D-DESS images compared to the original 2D TSE images. CONCLUSION The proposed method can successfully synthesize 3D DESS images from 2D TSE images to provide images suitable for automated cartilage segmentation.
Collapse
Affiliation(s)
- Aleš Neubert
- The Australian e-Health Research Centre, CSIRO Health and Biosecurity, Herston, Australia
| | - Pierrick Bourgeat
- The Australian e-Health Research Centre, CSIRO Health and Biosecurity, Herston, Australia
| | - Jason Wood
- The Australian e-Health Research Centre, CSIRO Health and Biosecurity, Herston, Australia
| | - Craig Engstrom
- School of Information Technology and Electrical Engineering, The University of Queensland, St. Lucia, Australia
| | - Shekhar S Chandra
- School of Information Technology and Electrical Engineering, The University of Queensland, St. Lucia, Australia
| | - Stuart Crozier
- School of Information Technology and Electrical Engineering, The University of Queensland, St. Lucia, Australia
| | - Jurgen Fripp
- The Australian e-Health Research Centre, CSIRO Health and Biosecurity, Herston, Australia
| |
Collapse
|
24
|
Huang Y, Zheng F, Cong R, Huang W, Scott MR, Shao L. MCMT-GAN: Multi-Task Coherent Modality Transferable GAN for 3D Brain Image Synthesis. IEEE TRANSACTIONS ON IMAGE PROCESSING : A PUBLICATION OF THE IEEE SIGNAL PROCESSING SOCIETY 2020; PP:8187-8198. [PMID: 32746245 DOI: 10.1109/tip.2020.3011557] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/26/2023]
Abstract
The ability to synthesize multi-modality data is highly desirable for many computer-aided medical applications, e.g. clinical diagnosis and neuroscience research, since rich imaging cohorts offer diverse and complementary information unraveling human tissues. However, collecting acquisitions can be limited by adversary factors such as patient discomfort, expensive cost and scanner unavailability. In this paper, we propose a multi-task coherent modality transferable GAN (MCMT-GAN) to address this issue for brain MRI synthesis in an unsupervised manner. Through combining the bidirectional adversarial loss, cycle-consistency loss, domain adapted loss and manifold regularization in a volumetric space, MCMT-GAN is robust for multi-modality brain image synthesis with visually high fidelity. In addition, we complement discriminators collaboratively working with segmentors which ensure the usefulness of our results to segmentation task. Experiments evaluated on various cross-modality synthesis show that our method produces visually impressive results with substitutability for clinical post-processing and also exceeds the state-of-the-art methods.
Collapse
|
25
|
Kim S, Jang H, Jang J, Lee YH, Hwang D. Deep‐learned short tau inversion recovery imaging using multi‐contrast MR images. Magn Reson Med 2020; 84:2994-3008. [DOI: 10.1002/mrm.28327] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/01/2020] [Revised: 04/27/2020] [Accepted: 04/27/2020] [Indexed: 01/22/2023]
Affiliation(s)
- Sewon Kim
- School of Electrical and Electronic Engineering Yonsei University Seoul Korea
| | - Hanbyol Jang
- School of Electrical and Electronic Engineering Yonsei University Seoul Korea
| | - Jinseong Jang
- School of Electrical and Electronic Engineering Yonsei University Seoul Korea
| | - Young Han Lee
- Department of Radiology and Center for Clinical Imaging Data Science (CCIDS) Yonsei University College of Medicine Seoul Korea
| | - Dosik Hwang
- School of Electrical and Electronic Engineering Yonsei University Seoul Korea
| |
Collapse
|
26
|
Carass A, Roy S, Gherman A, Reinhold JC, Jesson A, Arbel T, Maier O, Handels H, Ghafoorian M, Platel B, Birenbaum A, Greenspan H, Pham DL, Crainiceanu CM, Calabresi PA, Prince JL, Roncal WRG, Shinohara RT, Oguz I. Evaluating White Matter Lesion Segmentations with Refined Sørensen-Dice Analysis. Sci Rep 2020; 10:8242. [PMID: 32427874 PMCID: PMC7237671 DOI: 10.1038/s41598-020-64803-w] [Citation(s) in RCA: 56] [Impact Index Per Article: 14.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/03/2019] [Accepted: 04/20/2020] [Indexed: 11/09/2022] Open
Abstract
The Sørensen-Dice index (SDI) is a widely used measure for evaluating medical image segmentation algorithms. It offers a standardized measure of segmentation accuracy which has proven useful. However, it offers diminishing insight when the number of objects is unknown, such as in white matter lesion segmentation of multiple sclerosis (MS) patients. We present a refinement for finer grained parsing of SDI results in situations where the number of objects is unknown. We explore these ideas with two case studies showing what can be learned from our two presented studies. Our first study explores an inter-rater comparison, showing that smaller lesions cannot be reliably identified. In our second case study, we demonstrate fusing multiple MS lesion segmentation algorithms based on the insights into the algorithms provided by our analysis to generate a segmentation that exhibits improved performance. This work demonstrates the wealth of information that can be learned from refined analysis of medical image segmentations.
Collapse
Affiliation(s)
- Aaron Carass
- Department of Electrical and Computer Engineering, The Johns Hopkins University, Baltimore, MD, 21218, USA.
- Department of Computer Science, The Johns Hopkins University, Baltimore, MD, 21218, USA.
| | - Snehashis Roy
- CNRM, The Henry M. Jackson Foundation for the Advancement of Military Medicine, Bethesda, MD, 20817, USA
| | - Adrian Gherman
- Department of Biostatistics, The Johns Hopkins University, Baltimore, MD, 21205, USA
| | - Jacob C Reinhold
- Department of Electrical and Computer Engineering, The Johns Hopkins University, Baltimore, MD, 21218, USA
| | - Andrew Jesson
- Centre For Intelligent Machines, McGill University, Montréal, QC, H3A 0E9, Canada
| | - Tal Arbel
- Centre For Intelligent Machines, McGill University, Montréal, QC, H3A 0E9, Canada
| | - Oskar Maier
- Institute of Medical Informatics, University of Lübeck, 23538, Lübeck, Germany
| | - Heinz Handels
- Institute of Medical Informatics, University of Lübeck, 23538, Lübeck, Germany
| | - Mohsen Ghafoorian
- Institute for Computing and Information Sciences, Radboud University, 6525, HP, Nijmegen, Netherlands
| | - Bram Platel
- Diagnostic Image Analysis Group, Radboud University Medical Center, 6525, GA, Nijmegen, Netherlands
| | - Ariel Birenbaum
- Department of Electrical Engineering, Tel-Aviv University, Tel-Aviv, 69978, Israel
| | - Hayit Greenspan
- Department of Biomedical Engineering, Tel-Aviv University, Tel-Aviv, 69978, Israel
| | - Dzung L Pham
- CNRM, The Henry M. Jackson Foundation for the Advancement of Military Medicine, Bethesda, MD, 20817, USA
| | - Ciprian M Crainiceanu
- Department of Biostatistics, The Johns Hopkins University, Baltimore, MD, 21205, USA
| | - Peter A Calabresi
- Department of Neurology, The Johns Hopkins University School of Medicine, Baltimore, MD, 21287, USA
| | - Jerry L Prince
- Department of Electrical and Computer Engineering, The Johns Hopkins University, Baltimore, MD, 21218, USA
- Department of Computer Science, The Johns Hopkins University, Baltimore, MD, 21218, USA
| | - William R Gray Roncal
- Department of Computer Science, The Johns Hopkins University, Baltimore, MD, 21218, USA
| | - Russell T Shinohara
- Penn Statistics in Imaging and Visualization Center, Department of Biostatistics & Epidemiology, University of Pennsylvania, Philadelphia, PA, 19104, USA
- Center for Biomedical Image Computing and Analytics, Department of Radiology, University of Pennsylvania, Philadelphia, PA, 19104, USA
| | - Ipek Oguz
- Department of Electrical Engineering and Computer Science, Vanderbilt University, Nashville, TN, 37203, USA
| |
Collapse
|
27
|
Yang Q, Li N, Zhao Z, Fan X, Chang EIC, Xu Y. MRI Cross-Modality Image-to-Image Translation. Sci Rep 2020; 10:3753. [PMID: 32111966 PMCID: PMC7048849 DOI: 10.1038/s41598-020-60520-6] [Citation(s) in RCA: 26] [Impact Index Per Article: 6.5] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/09/2019] [Accepted: 02/12/2020] [Indexed: 11/23/2022] Open
Abstract
We present a cross-modality generation framework that learns to generate translated modalities from given modalities in MR images. Our proposed method performs Image Modality Translation (abbreviated as IMT) by means of a deep learning model that leverages conditional generative adversarial networks (cGANs). Our framework jointly exploits the low-level features (pixel-wise information) and high-level representations (e.g. brain tumors, brain structure like gray matter, etc.) between cross modalities which are important for resolving the challenging complexity in brain structures. Our framework can serve as an auxiliary method in medical use and has great application potential. Based on our proposed framework, we first propose a method for cross-modality registration by fusing the deformation fields to adopt the cross-modality information from translated modalities. Second, we propose an approach for MRI segmentation, translated multichannel segmentation (TMS), where given modalities, along with translated modalities, are segmented by fully convolutional networks (FCN) in a multichannel manner. Both of these two methods successfully adopt the cross-modality information to improve the performance without adding any extra data. Experiments demonstrate that our proposed framework advances the state-of-the-art on five brain MRI datasets. We also observe encouraging results in cross-modality registration and segmentation on some widely adopted brain datasets. Overall, our work can serve as an auxiliary method in medical use and be applied to various tasks in medical fields.
Collapse
Grants
- This work is supported by Microsoft Research under the eHealth program, the National Natural Science Foundation in China under Grant 81771910, the National Science and Technology Major Project of the Ministry of Science and Technology in China under Grant 2017YFC0110903, the Beijing Natural Science Foundation in China under Grant 4152033, the Technology and Innovation Commission of Shenzhen in China under Grant shenfagai2016-627, Beijing Young Talent Project in China, the Fundamental Research Funds for the Central Universities of China under Grant SKLSDE-2017ZX-08 from the State Key Laboratory of Software Development Environment in Beihang University in China, the 111 Project in China under Grant B13003.
- This work is supported by the National Science and Technology Major Project of the Ministry of Science and Technology in China under Grant 2017YFC0110903, Microsoft Research under the eHealth program, the National Natural Science Foundation in China under Grant 81771910, the Beijing Natural Science Foundation in China under Grant 4152033, the Technology and Innovation Commission of Shenzhen in China under Grant shenfagai2016-627, Beijing Young Talent Project in China, the Fundamental Research Funds for the Central Universities of China under Grant SKLSDE-2017ZX-08 from the State Key Laboratory of Software Development Environment in Beihang University in China, the 111 Project in China under Grant B13003.
Collapse
Affiliation(s)
- Qianye Yang
- State Key Laboratory of Software Development Environment and Key Laboratory of Biomechanics and Mechanobiology of Ministry of Education and Research Institute of Beihang University in Shenzhen, Beijing Advanced Innovation Center for Biomedical Engineering, Beihang University, Beijing, 100191, China
| | - Nannan Li
- State Key Laboratory of Software Development Environment and Key Laboratory of Biomechanics and Mechanobiology of Ministry of Education and Research Institute of Beihang University in Shenzhen, Beijing Advanced Innovation Center for Biomedical Engineering, Beihang University, Beijing, 100191, China
- Ping An Technology (Shenzhen) Co., Ltd., Shanghai, 200030, China
| | - Zixu Zhao
- State Key Laboratory of Software Development Environment and Key Laboratory of Biomechanics and Mechanobiology of Ministry of Education and Research Institute of Beihang University in Shenzhen, Beijing Advanced Innovation Center for Biomedical Engineering, Beihang University, Beijing, 100191, China
| | - Xingyu Fan
- Bioengineering College of Chongqing University, Chongqing, 400044, China
| | | | - Yan Xu
- State Key Laboratory of Software Development Environment and Key Laboratory of Biomechanics and Mechanobiology of Ministry of Education and Research Institute of Beihang University in Shenzhen, Beijing Advanced Innovation Center for Biomedical Engineering, Beihang University, Beijing, 100191, China.
- Microsoft Research Asia, Beijing, 100080, China.
| |
Collapse
|
28
|
Eslami M, Tabarestani S, Albarqouni S, Adeli E, Navab N, Adjouadi M. Image-to-Images Translation for Multi-Task Organ Segmentation and Bone Suppression in Chest X-Ray Radiography. IEEE TRANSACTIONS ON MEDICAL IMAGING 2020; 39:2553-2565. [PMID: 32078541 DOI: 10.1109/tmi.2020.2974159] [Citation(s) in RCA: 18] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/26/2023]
Abstract
Chest X-ray radiography is one of the earliest medical imaging technologies and remains one of the most widely-used for diagnosis, screening, and treatment follow up of diseases related to lungs and heart. The literature in this field of research reports many interesting studies dealing with the challenging tasks of bone suppression and organ segmentation but performed separately, limiting any learning that comes with the consolidation of parameters that could optimize both processes. This study, and for the first time, introduces a multitask deep learning model that generates simultaneously the bone-suppressed image and the organ-segmented image, enhancing the accuracy of tasks, minimizing the number of parameters needed by the model and optimizing the processing time, all by exploiting the interplay between the network parameters to benefit the performance of both tasks. The architectural design of this model, which relies on a conditional generative adversarial network, reveals the process on how the wellestablished pix2pix network (image-to-image network) is modified to fit the need for multitasking and extending it to the new image-to-images architecture. The developed source code of this multitask model is shared publicly on Github as the first attempt for providing the two-task pix2pix extension, a supervised/paired/aligned/registered image-to-images translation which would be useful in many multitask applications. Dilated convolutions are also used to improve the results through a more effective receptive field assessment. The comparison with state-of-the-art al-gorithms along with ablation study and a demonstration video1 are provided to evaluate the efficacy and gauge the merits of the proposed approach.
Collapse
|
29
|
Model-Driven Deep Attention Network for Ultra-fast Compressive Sensing MRI Guided by Cross-contrast MR Image. MEDICAL IMAGE COMPUTING AND COMPUTER ASSISTED INTERVENTION – MICCAI 2020 2020. [DOI: 10.1007/978-3-030-59713-9_19] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/13/2022]
|
30
|
Dar SU, Yurt M, Karacan L, Erdem A, Erdem E, Cukur T. Image Synthesis in Multi-Contrast MRI With Conditional Generative Adversarial Networks. IEEE TRANSACTIONS ON MEDICAL IMAGING 2019; 38:2375-2388. [PMID: 30835216 DOI: 10.1109/tmi.2019.2901750] [Citation(s) in RCA: 207] [Impact Index Per Article: 41.4] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/09/2023]
Abstract
Acquiring images of the same anatomy with multiple different contrasts increases the diversity of diagnostic information available in an MR exam. Yet, the scan time limitations may prohibit the acquisition of certain contrasts, and some contrasts may be corrupted by noise and artifacts. In such cases, the ability to synthesize unacquired or corrupted contrasts can improve diagnostic utility. For multi-contrast synthesis, the current methods learn a nonlinear intensity transformation between the source and target images, either via nonlinear regression or deterministic neural networks. These methods can, in turn, suffer from the loss of structural details in synthesized images. Here, in this paper, we propose a new approach for multi-contrast MRI synthesis based on conditional generative adversarial networks. The proposed approach preserves intermediate-to-high frequency details via an adversarial loss, and it offers enhanced synthesis performance via pixel-wise and perceptual losses for registered multi-contrast images and a cycle-consistency loss for unregistered images. Information from neighboring cross-sections are utilized to further improve synthesis quality. Demonstrations on T1- and T2- weighted images from healthy subjects and patients clearly indicate the superior performance of the proposed approach compared to the previous state-of-the-art methods. Our synthesis approach can help improve the quality and versatility of the multi-contrast MRI exams without the need for prolonged or repeated examinations.
Collapse
|
31
|
Zöllei L, Jaimes C, Saliba E, Grant PE, Yendiki A. TRActs constrained by UnderLying INfant anatomy (TRACULInA): An automated probabilistic tractography tool with anatomical priors for use in the newborn brain. Neuroimage 2019; 199:1-17. [PMID: 31132451 PMCID: PMC6688923 DOI: 10.1016/j.neuroimage.2019.05.051] [Citation(s) in RCA: 11] [Impact Index Per Article: 2.2] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/07/2018] [Revised: 05/14/2019] [Accepted: 05/18/2019] [Indexed: 10/26/2022] Open
Abstract
The ongoing myelination of white-matter fiber bundles plays a significant role in brain development. However, reliable and consistent identification of these bundles from infant brain MRIs is often challenging due to inherently low diffusion anisotropy, as well as motion and other artifacts. In this paper we introduce a new tool for automated probabilistic tractography specifically designed for newborn infants. Our tool incorporates prior information about the anatomical neighborhood of white-matter pathways from a training data set. In our experiments, we evaluate this tool on data from both full-term and prematurely born infants and demonstrate that it can reconstruct known white-matter tracts in both groups robustly, even in the presence of differences between the training set and study subjects. Additionally, we evaluate it on a publicly available large data set of healthy term infants (UNC Early Brain Development Program). This paves the way for performing a host of sophisticated analyses in newborns that we have previously implemented for the adult brain, such as pointwise analysis along tracts and longitudinal analysis, in both health and disease.
Collapse
Affiliation(s)
- Lilla Zöllei
- Massachusetts General Hospital, Boston, United States.
| | | | | | | | | |
Collapse
|
32
|
Jog A, Hoopes A, Greve DN, Van Leemput K, Fischl B. PSACNN: Pulse sequence adaptive fast whole brain segmentation. Neuroimage 2019; 199:553-569. [PMID: 31129303 PMCID: PMC6688920 DOI: 10.1016/j.neuroimage.2019.05.033] [Citation(s) in RCA: 17] [Impact Index Per Article: 3.4] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/17/2019] [Revised: 05/09/2019] [Accepted: 05/12/2019] [Indexed: 01/07/2023] Open
Abstract
With the advent of convolutional neural networks (CNN), supervised learning methods are increasingly being used for whole brain segmentation. However, a large, manually annotated training dataset of labeled brain images required to train such supervised methods is frequently difficult to obtain or create. In addition, existing training datasets are generally acquired with a homogeneous magnetic resonance imaging (MRI) acquisition protocol. CNNs trained on such datasets are unable to generalize on test data with different acquisition protocols. Modern neuroimaging studies and clinical trials are necessarily multi-center initiatives with a wide variety of acquisition protocols. Despite stringent protocol harmonization practices, it is very difficult to standardize the gamut of MRI imaging parameters across scanners, field strengths, receive coils etc., that affect image contrast. In this paper we propose a CNN-based segmentation algorithm that, in addition to being highly accurate and fast, is also resilient to variation in the input acquisition. Our approach relies on building approximate forward models of pulse sequences that produce a typical test image. For a given pulse sequence, we use its forward model to generate plausible, synthetic training examples that appear as if they were acquired in a scanner with that pulse sequence. Sampling over a wide variety of pulse sequences results in a wide variety of augmented training examples that help build an image contrast invariant model. Our method trains a single CNN that can segment input MRI images with acquisition parameters as disparate as T1-weighted and T2-weighted contrasts with only T1-weighted training data. The segmentations generated are highly accurate with state-of-the-art results (overall Dice overlap=0.94), with a fast run time (≈ 45 s), and consistent across a wide range of acquisition protocols.
Collapse
Affiliation(s)
- Amod Jog
- Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology, Massachusetts General Hospital, Charlestown, MA, 02129, United States; Department of Radiology, Harvard Medical School, United States.
| | - Andrew Hoopes
- Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology, Massachusetts General Hospital, Charlestown, MA, 02129, United States
| | - Douglas N Greve
- Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology, Massachusetts General Hospital, Charlestown, MA, 02129, United States; Department of Radiology, Harvard Medical School, United States
| | - Koen Van Leemput
- Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology, Massachusetts General Hospital, Charlestown, MA, 02129, United States; Department of Health Technology, Technical University of Denmark, Denmark
| | - Bruce Fischl
- Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology, Massachusetts General Hospital, Charlestown, MA, 02129, United States; Department of Radiology, Harvard Medical School, United States; Division of Health Sciences and Technology and Engineering and Computer Science MIT, Cambridge, MA, United States
| |
Collapse
|
33
|
Yu B, Zhou L, Wang L, Shi Y, Fripp J, Bourgeat P. Ea-GANs: Edge-Aware Generative Adversarial Networks for Cross-Modality MR Image Synthesis. IEEE TRANSACTIONS ON MEDICAL IMAGING 2019; 38:1750-1762. [PMID: 30714911 DOI: 10.1109/tmi.2019.2895894] [Citation(s) in RCA: 96] [Impact Index Per Article: 19.2] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/09/2023]
Abstract
Magnetic resonance (MR) imaging is a widely used medical imaging protocol that can be configured to provide different contrasts between the tissues in human body. By setting different scanning parameters, each MR imaging modality reflects the unique visual characteristic of scanned body part, benefiting the subsequent analysis from multiple perspectives. To utilize the complementary information from multiple imaging modalities, cross-modality MR image synthesis has aroused increasing research interest recently. However, most existing methods only focus on minimizing pixel/voxel-wise intensity difference but ignore the textural details of image content structure, which affects the quality of synthesized images. In this paper, we propose edge-aware generative adversarial networks (Ea-GANs) for cross-modality MR image synthesis. Specifically, we integrate edge information, which reflects the textural structure of image content and depicts the boundaries of different objects in images, to reduce this gap. Corresponding to different learning strategies, two frameworks are proposed, i.e., a generator-induced Ea-GAN (gEa-GAN) and a discriminator-induced Ea-GAN (dEa-GAN). The gEa-GAN incorporates the edge information via its generator, while the dEa-GAN further does this from both the generator and the discriminator so that the edge similarity is also adversarially learned. In addition, the proposed Ea-GANs are 3D-based and utilize hierarchical features to capture contextual information. The experimental results demonstrate that the proposed Ea-GANs, especially the dEa-GAN, outperform multiple state-of-the-art methods for cross-modality MR image synthesis in both qualitative and quantitative measures. Moreover, the dEa-GAN also shows excellent generality to generic image synthesis tasks on benchmark datasets about facades, maps, and cityscapes.
Collapse
|
34
|
Dual-domain convolutional neural networks for improving structural information in 3 T MRI. Magn Reson Imaging 2019; 64:90-100. [PMID: 31175927 DOI: 10.1016/j.mri.2019.05.023] [Citation(s) in RCA: 15] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/14/2018] [Revised: 05/19/2019] [Accepted: 05/19/2019] [Indexed: 12/11/2022]
Abstract
We propose a novel dual-domain convolutional neural network framework to improve structural information of routine 3 T images. We introduce a parameter-efficient butterfly network that involves two complementary domains: a spatial domain and a frequency domain. The butterfly network allows the interaction of these two domains in learning the complex mapping from 3 T to 7 T images. We verified the efficacy of the dual-domain strategy and butterfly network using 3 T and 7 T image pairs. Experimental results demonstrate that the proposed framework generates synthetic 7 T-like images and achieves performance superior to state-of-the-art methods.
Collapse
|
35
|
Hazra A, Reich BJ, Reich DS, Shinohara RT, Staicu AM. A Spatio-Temporal Model for Longitudinal Image-on-Image Regression. STATISTICS IN BIOSCIENCES 2019; 11:22-46. [PMID: 31156722 PMCID: PMC6537615] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 06/09/2023]
Abstract
Neurologists and radiologists often use magnetic resonance imaging (MRI) in the management of subjects with multiple sclerosis (MS) because it is sensitive to inflammatory and demyelinative changes in the white matter of the brain and spinal cord. Two conventional modalities used for identifying lesions are T1-weighted (T1) and T2-weighted fluid-attenuated inversion recovery (FLAIR) imaging, which are used clinically and in research studies. Magnetization transfer ratio (MTR), which is available only in research settings, is an advanced MRI modality that has been used extensively for measuring disease-related demyelination both in white matter lesions as well across normal-appearing white matter. Acquiring MTR is not standard in clinical practice, due to the increased scan time and cost. Hence, prediction of MTR based on the modalities T1 and FLAIR could have great impact on the availability of these promising measures for improved patient management. We propose a spatio-temporal regression model for image response and image predictors that are acquired longitudinally, with images being co-registered within the subject but not across subjects. The model is additive, with the response at a voxel being dependent on the available covariates not only through the current voxel but also on the imaging information from the voxels within a neighboring spatial region as well as their temporal gradients. We propose a dynamic Bayesian estimation procedure that updates the parameters of the subject-specific regression model as data accummulates. To bypass the computational challenges associated with a Bayesian approach for high-dimensional imaging data, we propose an approximate Bayesian inference technique. We assess the model fitting and the prediction performance using longitudinally acquired MRI images from 46 MS patients.
Collapse
Affiliation(s)
- Arnab Hazra
- North Carolina State University, Raleigh, NC, USA
| | | | - Daniel S Reich
- National Institute of Neurological Disorders and Stroke, Bethesda, MD, USA
| | | | | |
Collapse
|
36
|
Reinhold JC, Dewey BE, Carass A, Prince JL. Evaluating the Impact of Intensity Normalization on MR Image Synthesis. PROCEEDINGS OF SPIE--THE INTERNATIONAL SOCIETY FOR OPTICAL ENGINEERING 2019; 10949. [PMID: 31551645 DOI: 10.1117/12.2513089] [Citation(s) in RCA: 54] [Impact Index Per Article: 10.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/14/2022]
Abstract
Image synthesis learns a transformation from the intensity features of an input image to yield a different tissue contrast of the output image. This process has been shown to have application in many medical image analysis tasks including imputation, registration, and segmentation. To carry out synthesis, the intensities of the input images are typically scaled-i.e., normalized-both in training to learn the transformation and in testing when applying the transformation, but it is not presently known what type of input scaling is optimal. In this paper, we consider seven different intensity normalization algorithms and three different synthesis methods to evaluate the impact of normalization. Our experiments demonstrate that intensity normalization as a preprocessing step improves the synthesis results across all investigated synthesis algorithms. Furthermore, we show evidence that suggests intensity normalization is vital for successful deep learning-based MR image synthesis.
Collapse
Affiliation(s)
- Jacob C Reinhold
- Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, MD, USA 21218
| | - Blake E Dewey
- Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, MD, USA 21218.,F.M. Kirby Center for Functional Brain Imaging, Kennedy Krieger Institute, Baltimore, MD, USA, 21205
| | - Aaron Carass
- Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, MD, USA 21218.,Department of Computer Science, Johns Hopkins University, Baltimore, MD, USA 21218
| | - Jerry L Prince
- Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, MD, USA 21218.,Department of Computer Science, Johns Hopkins University, Baltimore, MD, USA 21218
| |
Collapse
|
37
|
Huo Y, Xu Z, Moon H, Bao S, Assad A, Moyo TK, Savona MR, Abramson RG, Landman BA. SynSeg-Net: Synthetic Segmentation Without Target Modality Ground Truth. IEEE TRANSACTIONS ON MEDICAL IMAGING 2018; 38:10.1109/TMI.2018.2876633. [PMID: 30334788 PMCID: PMC6504618 DOI: 10.1109/tmi.2018.2876633] [Citation(s) in RCA: 105] [Impact Index Per Article: 17.5] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/18/2023]
Abstract
A key limitation of deep convolutional neural networks (DCNN) based image segmentation methods is the lack of generalizability. Manually traced training images are typically required when segmenting organs in a new imaging modality or from distinct disease cohort. The manual efforts can be alleviated if the manually traced images in one imaging modality (e.g., MRI) are able to train a segmentation network for another imaging modality (e.g., CT). In this paper, we propose an end-to-end synthetic segmentation network (SynSeg-Net) to train a segmentation network for a target imaging modality without having manual labels. SynSeg-Net is trained by using (1) unpaired intensity images from source and target modalities, and (2) manual labels only from source modality. SynSeg-Net is enabled by the recent advances of cycle generative adversarial networks (CycleGAN) and DCNN. We evaluate the performance of the SynSeg-Net on two experiments: (1) MRI to CT splenomegaly synthetic segmentation for abdominal images, and (2) CT to MRI total intracranial volume synthetic segmentation (TICV) for brain images. The proposed end-to-end approach achieved superior performance to two stage methods. Moreover, the SynSeg-Net achieved comparable performance to the traditional segmentation network using target modality labels in certain scenarios. The source code of SynSeg-Net is publicly available 2.
Collapse
|
38
|
Yang H, Sun J, Li H, Wang L, Xu Z. Neural multi-atlas label fusion: Application to cardiac MR images. Med Image Anal 2018; 49:60-75. [DOI: 10.1016/j.media.2018.07.009] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/03/2017] [Revised: 07/10/2018] [Accepted: 07/30/2018] [Indexed: 10/28/2022]
|
39
|
Iglesias JE, Modat M, Peter L, Stevens A, Annunziata R, Vercauteren T, Lein E, Fischl B, Ourselin S. Joint registration and synthesis using a probabilistic model for alignment of MRI and histological sections. Med Image Anal 2018; 50:127-144. [PMID: 30282061 PMCID: PMC6742511 DOI: 10.1016/j.media.2018.09.002] [Citation(s) in RCA: 16] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/17/2017] [Revised: 08/30/2018] [Accepted: 09/05/2018] [Indexed: 11/30/2022]
Abstract
Nonlinear registration of 2D histological sections with corresponding slices of MRI data is a critical step of 3D histology reconstruction algorithms. This registration is difficult due to the large differences in image contrast and resolution, as well as the complex nonrigid deformations and artefacts produced when sectioning the sample and mounting it on the glass slide. It has been shown in brain MRI registration that better spatial alignment across modalities can be obtained by synthesising one modality from the other and then using intra-modality registration metrics, rather than by using information theory based metrics to solve the problem directly. However, such an approach typically requires a database of aligned images from the two modalities, which is very difficult to obtain for histology and MRI. Here, we overcome this limitation with a probabilistic method that simultaneously solves for deformable registration and synthesis directly on the target images, without requiring any training data. The method is based on a probabilistic model in which the MRI slice is assumed to be a contrast-warped, spatially deformed version of the histological section. We use approximate Bayesian inference to iteratively refine the probabilistic estimate of the synthesis and the registration, while accounting for each other’s uncertainty. Moreover, manually placed landmarks can be seamlessly integrated in the framework for increased performance and robustness. Experiments on a synthetic dataset of MRI slices show that, compared with mutual information based registration, the proposed method makes it possible to use a much more flexible deformation model in the registration to improve its accuracy, without compromising robustness. Moreover, our framework also exploits information in manually placed landmarks more efficiently than mutual information: landmarks constrain the deformation field in both methods, but in our algorithm, it also has a positive effect on the synthesis – which further improves the registration. We also show results on two real, publicly available datasets: the Allen and BigBrain atlases. In both of them, the proposed method provides a clear improvement over mutual information based registration, both qualitatively (visual inspection) and quantitatively (registration error measured with pairs of manually annotated landmarks).
Collapse
Affiliation(s)
- Juan Eugenio Iglesias
- Translational Imaging Group, Centre for Medical Image Computing, University College London, UK.
| | - Marc Modat
- Translational Imaging Group, Centre for Medical Image Computing, University College London, UK
| | - Loïc Peter
- Wellcome EPSRC Centre for Interventional and Surgical Sciences (WEISS), University College London, UK
| | - Allison Stevens
- Martinos Center for Biomedical Imaging, Harvard Medical School and Massachusetts General Hospital, USA
| | - Roberto Annunziata
- Translational Imaging Group, Centre for Medical Image Computing, University College London, UK
| | - Tom Vercauteren
- Wellcome EPSRC Centre for Interventional and Surgical Sciences (WEISS), University College London, UK
| | - Ed Lein
- Allen Institute for Brain Science, USA
| | - Bruce Fischl
- Martinos Center for Biomedical Imaging, Harvard Medical School and Massachusetts General Hospital, USA; Computer Science and AI lab, Massachusetts Institute of Technology, USA
| | - Sebastien Ourselin
- Wellcome EPSRC Centre for Interventional and Surgical Sciences (WEISS), University College London, UK
| | | |
Collapse
|
40
|
Zhang Y, Cheng JZ, Xiang L, Yap PT, Shen D. Dual-Domain Cascaded Regression for Synthesizing 7T from 3T MRI. MEDICAL IMAGE COMPUTING AND COMPUTER-ASSISTED INTERVENTION : MICCAI ... INTERNATIONAL CONFERENCE ON MEDICAL IMAGE COMPUTING AND COMPUTER-ASSISTED INTERVENTION 2018; 11070:410-417. [PMID: 30957107 PMCID: PMC6448783 DOI: 10.1007/978-3-030-00928-1_47] [Citation(s) in RCA: 8] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 01/22/2023]
Abstract
Due to the high cost and low accessibility of 7T magnetic resonance imaging (MRI) scanners, we propose a novel dual-domain cascaded regression framework to synthesize 7T images from the routine 3T images. Our framework is composed of two parallel and interactive multi-stage regression streams, where one stream regresses on spatial domain and the other regresses on frequency domain. These two streams complement each other and enable the learning of complex mappings between 3T and 7T images. We evaluated the proposed framework on a set of 3T and 7T images by leave-one-out cross-validation. Experimental results demonstrate that the proposed framework generates realistic 7T images and achieves better results than state-of-the-art methods.
Collapse
Affiliation(s)
- Yongqin Zhang
- Department of Radiology and BRIC, University of North Carolina at Chapel Hill, Chapel Hill, NC 27599, USA
- School of Information Science and Technology, Northwest University, Xi'an 710127, China
| | - Jie-Zhi Cheng
- Shanghai United Imaging Intelligence Co., Ltd., Shanghai 201807, China
| | - Lei Xiang
- Department of Radiology and BRIC, University of North Carolina at Chapel Hill, Chapel Hill, NC 27599, USA
| | - Pew-Thian Yap
- Department of Radiology and BRIC, University of North Carolina at Chapel Hill, Chapel Hill, NC 27599, USA
| | - Dinggang Shen
- Department of Radiology and BRIC, University of North Carolina at Chapel Hill, Chapel Hill, NC 27599, USA
| |
Collapse
|
41
|
Chartsias A, Joyce T, Giuffrida MV, Tsaftaris SA. Multimodal MR Synthesis via Modality-Invariant Latent Representation. IEEE TRANSACTIONS ON MEDICAL IMAGING 2018; 37:803-814. [PMID: 29053447 PMCID: PMC5904017 DOI: 10.1109/tmi.2017.2764326] [Citation(s) in RCA: 116] [Impact Index Per Article: 19.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/22/2023]
Abstract
We propose a multi-input multi-output fully convolutional neural network model for MRI synthesis. The model is robust to missing data, as it benefits from, but does not require, additional input modalities. The model is trained end-to-end, and learns to embed all input modalities into a shared modality-invariant latent space. These latent representations are then combined into a single fused representation, which is transformed into the target output modality with a learnt decoder. We avoid the need for curriculum learning by exploiting the fact that the various input modalities are highly correlated. We also show that by incorporating information from segmentation masks the model can both decrease its error and generate data with synthetic lesions. We evaluate our model on the ISLES and BRATS data sets and demonstrate statistically significant improvements over state-of-the-art methods for single input tasks. This improvement increases further when multiple input modalities are used, demonstrating the benefits of learning a common latent space, again resulting in a statistically significant improvement over the current best method. Finally, we demonstrate our approach on non skull-stripped brain images, producing a statistically significant improvement over the previous best method. Code is made publicly available at https://github.com/agis85/multimodal_brain_synthesis.
Collapse
Affiliation(s)
| | | | - Mario Valerio Giuffrida
- School of Engineering at The University of Edinburgh. Giuffrida and Tsaftaris are also with The Alan Turing Institute of London. Giuffrida is also with IMT Lucca
| | - Sotirios A. Tsaftaris
- School of Engineering at The University of Edinburgh. Giuffrida and Tsaftaris are also with The Alan Turing Institute of London. Giuffrida is also with IMT Lucca
| |
Collapse
|
42
|
Hazra A, Reich BJ, Reich DS, Shinohara RT, Staicu AM. A Spatio-Temporal Model for Longitudinal Image-on-Image Regression. STATISTICS IN BIOSCIENCES 2017. [DOI: 10.1007/s12561-017-9206-z] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.1] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/31/2022]
|
43
|
Cao X, Yang J, Gao Y, Guo Y, Wu G, Shen D. Dual-core steered non-rigid registration for multi-modal images via bi-directional image synthesis. Med Image Anal 2017; 41:18-31. [PMID: 28533050 PMCID: PMC5896773 DOI: 10.1016/j.media.2017.05.004] [Citation(s) in RCA: 48] [Impact Index Per Article: 6.9] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/26/2017] [Revised: 05/05/2017] [Accepted: 05/09/2017] [Indexed: 12/20/2022]
Abstract
In prostate cancer radiotherapy, computed tomography (CT) is widely used for dose planning purposes. However, because CT has low soft tissue contrast, it makes manual contouring difficult for major pelvic organs. In contrast, magnetic resonance imaging (MRI) provides high soft tissue contrast, which makes it ideal for accurate manual contouring. Therefore, the contouring accuracy on CT can be significantly improved if the contours in MRI can be mapped to CT domain by registering MRI with CT of the same subject, which would eventually lead to high treatment efficacy. In this paper, we propose a bi-directional image synthesis based approach for MRI-to-CT pelvic image registration. First, we use patch-wise random forest with auto-context model to learn the appearance mapping from CT to MRI domain, and then vice versa. Consequently, we can synthesize a pseudo-MRI whose anatomical structures are exactly same with CT but with MRI-like appearance, and a pseudo-CT as well. Then, our MRI-to-CT registration can be steered in a dual manner, by simultaneously estimating two deformation pathways: 1) one from the pseudo-CT to the actual CT and 2) another from actual MRI to the pseudo-MRI. Next, a dual-core deformation fusion framework is developed to iteratively and effectively combine these two registration pathways by using complementary information from both modalities. Experiments on a dataset with real pelvic CT and MRI have shown improved registration performance of the proposed method by comparing it to the conventional registration methods, thus indicating its high potential of translation to the routine radiation therapy.
Collapse
Affiliation(s)
- Xiaohuan Cao
- School of Automation, Northwestern Polytechnical University, Xi'an, Shaanxi, China; Department of Radiology and BRIC, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA
| | - Jianhua Yang
- School of Automation, Northwestern Polytechnical University, Xi'an, Shaanxi, China
| | - Yaozong Gao
- Department of Radiology and BRIC, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA; Department of Computer Science, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA
| | - Yanrong Guo
- Department of Radiology and BRIC, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA
| | - Guorong Wu
- Department of Radiology and BRIC, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA
| | - Dinggang Shen
- Department of Radiology and BRIC, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA; Department of Brain and Cognitive Engineering, Korea University, Seoul 02841, Republic of Korea.
| |
Collapse
|
44
|
Bowles C, Qin C, Guerrero R, Gunn R, Hammers A, Dickie DA, Valdés Hernández M, Wardlaw J, Rueckert D. Brain lesion segmentation through image synthesis and outlier detection. Neuroimage Clin 2017; 16:643-658. [PMID: 29868438 PMCID: PMC5984574 DOI: 10.1016/j.nicl.2017.09.003] [Citation(s) in RCA: 17] [Impact Index Per Article: 2.4] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/23/2017] [Revised: 08/30/2017] [Accepted: 09/04/2017] [Indexed: 11/02/2022]
Abstract
Cerebral small vessel disease (SVD) can manifest in a number of ways. Many of these result in hyperintense regions visible on T2-weighted magnetic resonance (MR) images. The automatic segmentation of these lesions has been the focus of many studies. However, previous methods tended to be limited to certain types of pathology, as a consequence of either restricting the search to the white matter, or by training on an individual pathology. Here we present an unsupervised abnormality detection method which is able to detect abnormally hyperintense regions on FLAIR regardless of the underlying pathology or location. The method uses a combination of image synthesis, Gaussian mixture models and one class support vector machines, and needs only be trained on healthy tissue. We evaluate our method by comparing segmentation results from 127 subjects with SVD with three established methods and report significantly superior performance across a number of metrics.
Collapse
Affiliation(s)
| | - Chen Qin
- Department of Computing, Imperial College London, UK
| | | | - Roger Gunn
- Imanova Ltd., London, UK
- Department of Medicine, Imperial College London, UK
| | - Alexander Hammers
- Department of Computing, Imperial College London, UK
- King's College London & Guy's and St Thomas' PET Centre, Division of Imaging Sciences and Biomedical Engineering, St Thomas' Hospital, King's College London, UK
| | | | | | - Joanna Wardlaw
- Department of Neuroimaging Sciences, University of Edinburgh, UK
| | | |
Collapse
|
45
|
Bahrami K, Rekik I, Shi F, Shen D. Joint Reconstruction and Segmentation of 7T-like MR Images from 3T MRI Based on Cascaded Convolutional Neural Networks. ACTA ACUST UNITED AC 2017; 10433:764-772. [PMID: 30009281 DOI: 10.1007/978-3-319-66182-7_87] [Citation(s) in RCA: 6] [Impact Index Per Article: 0.9] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 04/22/2023]
Abstract
7T MRI scanner provides MR images with higher resolution and better contrast than 3T MR scanners. This helps many medical analysis tasks, including tissue segmentation. However, currently there is a very limited number of 7T MRI scanners worldwide. This motivates us to propose a novel image post-processing framework that can jointly generate high-resolution 7T-like images and their corresponding high-quality 7T-like tissue segmentation maps, solely from the routine 3T MR images. Our proposed framework comprises two parallel components, namely (1) reconstruction and (2) segmentation. The reconstruction component includes the multi-step cascaded convolutional neural networks (CNNs) that map the input 3T MR image to a 7T-like MR image, in terms of both resolution and contrast. Similarly, the segmentation component involves another paralleled cascaded CNNs, with a different architecture, to generate high-quality segmentation maps. These cascaded feedbacks between the two designed paralleled CNNs allow both tasks to mutually benefit from each another when learning the respective reconstruction and segmentation mappings. For evaluation, we have tested our framework on 15 subjects (with paired 3T and 7T images) using a leave-one-out cross-validation. The experimental results show that our estimated 7T-like images have richer anatomical details and better segmentation results, compared to the 3T MRI. Furthermore, our method also achieved better results in both reconstruction and segmentation tasks, compared to the state-of-the-art methods.
Collapse
Affiliation(s)
- Khosro Bahrami
- Department of Radiology and BRIC, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA
| | - Islem Rekik
- Department of Radiology and BRIC, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA
| | - Feng Shi
- Department of Radiology and BRIC, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA
| | - Dinggang Shen
- Department of Radiology and BRIC, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA
| |
Collapse
|
46
|
Zhao C, Carass A, Lee J, Jog A, Prince JL. A supervoxel based random forest synthesis framework for bidirectional MR/CT synthesis. SIMULATION AND SYNTHESIS IN MEDICAL IMAGING : ... INTERNATIONAL WORKSHOP, SASHIMI ..., HELD IN CONJUNCTION WITH MICCAI ..., PROCEEDINGS. SASHIMI (WORKSHOP) 2017; 10557:33-40. [PMID: 30221260 DOI: 10.1007/978-3-319-68127-6_4] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 01/06/2023]
Abstract
Synthesizing magnetic resonance (MR) and computed tomography (CT) images (from each other) has important implications for clinical neuroimaging. The MR to CT direction is critical for MRI-based radiotherapy planning and dose computation, whereas the CT to MR direction can provide an economic alternative to real MRI for image processing tasks. Additionally, synthesis in both directions can enhance MR/CT multi-modal image registration. Existing approaches have focused on synthesizing CT from MR. In this paper, we propose a multi-atlas based hybrid method to synthesize T1-weighted MR images from CT and CT images from T1-weighted MR images using a common framework. The task is carried out by: (a) computing a label field based on supervoxels for the subject image using joint label fusion; (b) correcting this result using a random forest classifier (RF-C); (c) spatial smoothing using a Markov random field; (d) synthesizing intensities using a set of RF regressors, one trained for each label. The algorithm is evaluated using a set of six registered CT and MR image pairs of the whole head.
Collapse
Affiliation(s)
- Can Zhao
- Dept. of Electrical and Computer Engineering, The Johns Hopkins University, Baltimore, MD 21218
| | - Aaron Carass
- Dept. of Electrical and Computer Engineering, The Johns Hopkins University, Baltimore, MD 21218
| | - Junghoon Lee
- Dept. of Radiation Oncology, The Johns Hopkins School of Medicine, Baltimore, MD 21287
| | - Amod Jog
- Dept. of Electrical and Computer Engineering, The Johns Hopkins University, Baltimore, MD 21218
| | - Jerry L Prince
- Dept. of Electrical and Computer Engineering, The Johns Hopkins University, Baltimore, MD 21218
| |
Collapse
|
47
|
Jiang Y, Liu F, Fan M, Li X, Zhao Z, Zeng Z, Wang Y, Xu D. Deducing magnetic resonance neuroimages based on knowledge from samples. Comput Med Imaging Graph 2017; 62:1-14. [PMID: 28807363 DOI: 10.1016/j.compmedimag.2017.07.005] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.1] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/21/2016] [Revised: 05/25/2017] [Accepted: 07/27/2017] [Indexed: 10/19/2022]
Abstract
PURPOSE Because individual variance always exists, using the same set of predetermined parameters for magnetic resonance imaging (MRI) may not be exactly suitable for each participant. We propose a knowledge-based method that can repair MRI data of undesired contrast as if a new scan were acquired using imaging parameters that had been individually optimized. METHODS The method employed a strategy called analogical reasoning to deduce voxel-wise relaxation properties using morphological and biological similarity. The proposed framework involves steps of intensity normalization, tissue segmentation, relaxation time deducing, and image deducing. RESULTS This approach has been preliminarily validated using conventional MRI data at 3T from several examples, including 5 normal and 9 clinical datasets. It can effectively improve the contrast of real MRI data by deducing imaging data using optimized imaging parameters based on deduced relaxation properties. The statistics of deduced images shows a high correlation with real data that were actually collected using the same set of imaging parameters. CONCLUSION The proposed method of deducing MRI data using knowledge of relaxation times alternatively provides a way of repairing MRI data of less optimal contrast. The method is also capable of optimizing an MRI protocol for individual participants, thereby realizing personalized MR imaging.
Collapse
Affiliation(s)
- Yuwei Jiang
- Shanghai Key Laboratory of Magnetic Resonance, MOE & Shanghai Key Laboratory of Brain Functional Genomics, Institute of Cognitive Neuroscience, East China Normal University, Shanghai 200062, PR China; Department of Psychiatry, Columbia University & Molecular Imaging and Neuropathology Division, New York State Psychiatric Institute, New York, 10032, USA
| | - Feng Liu
- Department of Psychiatry, Columbia University & Molecular Imaging and Neuropathology Division, New York State Psychiatric Institute, New York, 10032, USA
| | - Mingxia Fan
- Shanghai Key Laboratory of Magnetic Resonance, MOE & Shanghai Key Laboratory of Brain Functional Genomics, Institute of Cognitive Neuroscience, East China Normal University, Shanghai 200062, PR China
| | - Xuzhou Li
- Shanghai Key Laboratory of Magnetic Resonance, MOE & Shanghai Key Laboratory of Brain Functional Genomics, Institute of Cognitive Neuroscience, East China Normal University, Shanghai 200062, PR China; Department of Psychiatry, Columbia University & Molecular Imaging and Neuropathology Division, New York State Psychiatric Institute, New York, 10032, USA
| | - Zhiyong Zhao
- Shanghai Key Laboratory of Magnetic Resonance, MOE & Shanghai Key Laboratory of Brain Functional Genomics, Institute of Cognitive Neuroscience, East China Normal University, Shanghai 200062, PR China
| | - Zhaoling Zeng
- Shanghai University of Electric Power, Shanghai 200090, PR China
| | - Yi Wang
- MRI Research Institute, Radiology Department, Cornell University, New York, NY 10012, USA
| | - Dongrong Xu
- Department of Psychiatry, Columbia University & Molecular Imaging and Neuropathology Division, New York State Psychiatric Institute, New York, 10032, USA.
| |
Collapse
|
48
|
Bahrami K, Shi F, Rekik I, Gao Y, Shen D. 7T-guided super-resolution of 3T MRI. Med Phys 2017; 44:1661-1677. [PMID: 28177548 DOI: 10.1002/mp.12132] [Citation(s) in RCA: 33] [Impact Index Per Article: 4.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/08/2016] [Revised: 12/22/2016] [Accepted: 01/13/2017] [Indexed: 11/11/2022] Open
Abstract
PURPOSE High-resolution MR images can depict rich details of brain anatomical structures and show subtle changes in longitudinal data. 7T MRI scanners can acquire MR images with higher resolution and better tissue contrast than the routine 3T MRI scanners. However, 7T MRI scanners are currently more expensive and less available in clinical and research centers. To this end, we propose a method to generate super-resolution 3T MRI that resembles 7T MRI, which is called as 7T-like MR image in this paper. METHODS First, we propose a mapping from 3T MRI to 7T MRI space, using regression random forest. The mapped 3T MR images serve as intermediate results with similar appearance as 7T MR images. Second, we predict the final higher resolution 7T-like MR images based on sparse representation, using paired local dictionaries for both the mapped 3T MR images and 7T MR images. RESULTS Based on 15 subjects with both 3T and 7T MR images, the predicted 7T-like MR images by our method can best match the ground-truth 7T MR images, compared to other methods. Meanwhile, the experiment on brain tissue segmentation shows that our 7T-like MR images lead to the highest accuracy in the segmentation of WM, GM, and CSF brain tissues, compared to segmentations of 3T MR images as well as the reconstructed 7T-like MR images by other methods. CONCLUSIONS We propose a novel method for prediction of high-resolution 7T-like MR images from low-resolution 3T MR images. Our predicted 7T-like MR images demonstrate better spatial resolution compared to 3T MR images, as well as prediction results by other comparison methods. Such high-quality 7T-like MR images could better facilitate disease diagnosis and intervention.
Collapse
Affiliation(s)
- Khosro Bahrami
- Department of Radiology and BRIC, University of North Carolina at Chapel Hill, North Carolina, 27510, USA
| | - Feng Shi
- Department of Radiology and BRIC, University of North Carolina at Chapel Hill, North Carolina, 27510, USA
| | - Islem Rekik
- Department of Radiology and BRIC, University of North Carolina at Chapel Hill, North Carolina, 27510, USA
| | - Yaozong Gao
- Department of Radiology and BRIC, University of North Carolina at Chapel Hill, North Carolina, 27510, USA
| | - Dinggang Shen
- Department of Radiology and BRIC, University of North Carolina at Chapel Hill, North Carolina, 27510, USA.,Department of Brain and Cognitive Engineering, Korea University, Seoul, 02841, Republic of Korea
| |
Collapse
|
49
|
Chen M, Carass A, Jog A, Lee J, Roy S, Prince JL. Cross contrast multi-channel image registration using image synthesis for MR brain images. Med Image Anal 2017; 36:2-14. [PMID: 27816859 PMCID: PMC5239759 DOI: 10.1016/j.media.2016.10.005] [Citation(s) in RCA: 30] [Impact Index Per Article: 4.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/11/2015] [Revised: 10/13/2016] [Accepted: 10/17/2016] [Indexed: 11/21/2022]
Abstract
Multi-modal deformable registration is important for many medical image analysis tasks such as atlas alignment, image fusion, and distortion correction. Whereas a conventional method would register images with different modalities using modality independent features or information theoretic metrics such as mutual information, this paper presents a new framework that addresses the problem using a two-channel registration algorithm capable of using mono-modal similarity measures such as sum of squared differences or cross-correlation. To make it possible to use these same-modality measures, image synthesis is used to create proxy images for the opposite modality as well as intensity-normalized images from each of the two available images. The new deformable registration framework was evaluated by performing intra-subject deformation recovery, intra-subject boundary alignment, and inter-subject label transfer experiments using multi-contrast magnetic resonance brain imaging data. Three different multi-channel registration algorithms were evaluated, revealing that the framework is robust to the multi-channel deformable registration algorithm that is used. With a single exception, all results demonstrated improvements when compared against single channel registrations using the same algorithm with mutual information.
Collapse
Affiliation(s)
- Min Chen
- Department of Radiology, University of Pennsylvania, Philadelphia, PA 19104, USA.
| | - Aaron Carass
- Department of Electrical and Computer Engineering, The Johns Hopkins University, Baltimore, MD 21218, USA; Department of Computer Science, The Johns Hopkins University, Baltimore, MD 21218 USA.
| | - Amod Jog
- Department of Computer Science, The Johns Hopkins University, Baltimore, MD 21218 USA.
| | - Junghoon Lee
- Department of Electrical and Computer Engineering, The Johns Hopkins University, Baltimore, MD 21218, USA; Radiation Oncology and Molecular Radiation Sciences, The Johns Hopkins School of Medicine, Baltimore, MD 21287, USA.
| | - Snehashis Roy
- CNRM, The Henry M. Jackson Foundation for the Advancement of Military Medicine, Bethesda, MD 20892, USA.
| | - Jerry L Prince
- Department of Electrical and Computer Engineering, The Johns Hopkins University, Baltimore, MD 21218, USA; Department of Computer Science, The Johns Hopkins University, Baltimore, MD 21218 USA; Radiation Oncology and Molecular Radiation Sciences, The Johns Hopkins School of Medicine, Baltimore, MD 21287, USA.
| |
Collapse
|
50
|
Roy S, Butman JA, Pham DL. Robust skull stripping using multiple MR image contrasts insensitive to pathology. Neuroimage 2017; 146:132-147. [PMID: 27864083 PMCID: PMC5321800 DOI: 10.1016/j.neuroimage.2016.11.017] [Citation(s) in RCA: 64] [Impact Index Per Article: 9.1] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/28/2016] [Revised: 10/31/2016] [Accepted: 11/04/2016] [Indexed: 01/18/2023] Open
Abstract
Automatic skull-stripping or brain extraction of magnetic resonance (MR) images is often a fundamental step in many neuroimage processing pipelines. The accuracy of subsequent image processing relies on the accuracy of the skull-stripping. Although many automated stripping methods have been proposed in the past, it is still an active area of research particularly in the context of brain pathology. Most stripping methods are validated on T1-w MR images of normal brains, especially because high resolution T1-w sequences are widely acquired and ground truth manual brain mask segmentations are publicly available for normal brains. However, different MR acquisition protocols can provide complementary information about the brain tissues, which can be exploited for better distinction between brain, cerebrospinal fluid, and unwanted tissues such as skull, dura, marrow, or fat. This is especially true in the presence of pathology, where hemorrhages or other types of lesions can have similar intensities as skull in a T1-w image. In this paper, we propose a sparse patch based Multi-cONtrast brain STRipping method (MONSTR),2 where non-local patch information from one or more atlases, which contain multiple MR sequences and reference delineations of brain masks, are combined to generate a target brain mask. We compared MONSTR with four state-of-the-art, publicly available methods: BEaST, SPECTRE, ROBEX, and OptiBET. We evaluated the performance of these methods on 6 datasets consisting of both healthy subjects and patients with various pathologies. Three datasets (ADNI, MRBrainS, NAMIC) are publicly available, consisting of 44 healthy volunteers and 10 patients with schizophrenia. Other three in-house datasets, comprising 87 subjects in total, consisted of patients with mild to severe traumatic brain injury, brain tumors, and various movement disorders. A combination of T1-w, T2-w were used to skull-strip these datasets. We show significant improvement in stripping over the competing methods on both healthy and pathological brains. We also show that our multi-contrast framework is robust and maintains accurate performance across different types of acquisitions and scanners, even when using normal brains as atlases to strip pathological brains, demonstrating that our algorithm is applicable even when reference segmentations of pathological brains are not available to be used as atlases.
Collapse
Affiliation(s)
- Snehashis Roy
- Center for Neuroscience and Regenerative Medicine, Henry M. Jackson Foundation, United States.
| | - John A Butman
- Center for Neuroscience and Regenerative Medicine, Henry M. Jackson Foundation, United States; Diagnostic Radiology Department, National Institute of Health, United States
| | - Dzung L Pham
- Center for Neuroscience and Regenerative Medicine, Henry M. Jackson Foundation, United States
| |
Collapse
|