1
|
Sun Y, Liu X, Liu Y, Jin R, Pang Y. DIRECTION: Deep cascaded reconstruction residual-based feature modulation network for fast MRI reconstruction. Magn Reson Imaging 2024; 111:157-167. [PMID: 38642780 DOI: 10.1016/j.mri.2024.04.023] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/30/2023] [Revised: 02/24/2024] [Accepted: 04/14/2024] [Indexed: 04/22/2024]
Abstract
Deep cascaded networks have been extensively studied and applied to accelerate Magnetic Resonance Imaging (MRI) and have shown promising results. Most existing works employ a large cascading number for the sake of superior performances. However, due to the lack of proper guidance, the reconstruction performance can easily reach a plateau and even face degradation if simply increasing the cascading number. In this paper, we aim to boost the reconstruction performance from a novel perspective by proposing a parallel architecture called DIRECTION that fully exploits the guiding value of the reconstruction residual of each subnetwork. Specifically, we introduce a novel Reconstruction Residual-Based Feature Modulation Mechanism (RRFMM) which utilizes the reconstruction residual of the previous subnetwork to guide the next subnetwork at the feature level. To achieve this, a Residual Attention Modulation Block (RAMB) is proposed to generate attention maps using multi-scale residual features to modulate the image features of the corresponding scales. Equipped with this strategy, each subnetwork within the cascaded network possesses its unique optimization objective and emphasis rather than blindly updating its parameters. To further boost the performance, we introduce the Cross-Stage Feature Reuse Connection (CSFRC) and the Reconstruction Dense Connection (RDC), which can reduce information loss and enhance representative ability. We conduct sufficient experiments and evaluate our method on the fastMRI knee dataset using multiple subsampling masks. Comprehensive experimental results show that our method can markedly boost the performance of cascaded networks and significantly outperforms other compared state-of-the-art methods quantitatively and qualitatively.
Collapse
Affiliation(s)
- Yong Sun
- TJK-BIIT Lab, School of Electrical and Information Engineering, Tianjin 300072, China.
| | - Xiaohan Liu
- TJK-BIIT Lab, School of Electrical and Information Engineering, Tianjin 300072, China.
| | - Yiming Liu
- TJK-BIIT Lab, School of Electrical and Information Engineering, Tianjin 300072, China; Tiandatz Technology, Tianjin 301723, China.
| | - Ruiqi Jin
- TJK-BIIT Lab, School of Electrical and Information Engineering, Tianjin 300072, China.
| | - Yanwei Pang
- TJK-BIIT Lab, School of Electrical and Information Engineering, Tianjin 300072, China.
| |
Collapse
|
2
|
Siedler TM, Jakob PM, Herold V. Enhancing quality and speed in database-free neural network reconstructions of undersampled MRI with SCAMPI. Magn Reson Med 2024; 92:1232-1247. [PMID: 38748852 DOI: 10.1002/mrm.30114] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/24/2023] [Revised: 03/19/2024] [Accepted: 03/27/2024] [Indexed: 06/27/2024]
Abstract
PURPOSE We present SCAMPI (Sparsity Constrained Application of deep Magnetic resonance Priors for Image reconstruction), an untrained deep Neural Network for MRI reconstruction without previous training on datasets. It expands the Deep Image Prior approach with a multidomain, sparsity-enforcing loss function to achieve higher image quality at a faster convergence speed than previously reported methods. METHODS Two-dimensional MRI data from the FastMRI dataset with Cartesian undersampling in phase-encoding direction were reconstructed for different acceleration rates for single coil and multicoil data. RESULTS The performance of our architecture was compared to state-of-the-art Compressed Sensing methods and ConvDecoder, another untrained Neural Network for two-dimensional MRI reconstruction. SCAMPI outperforms these by better reducing undersampling artifacts and yielding lower error metrics in multicoil imaging. In comparison to ConvDecoder, the U-Net architecture combined with an elaborated loss-function allows for much faster convergence at higher image quality. SCAMPI can reconstruct multicoil data without explicit knowledge of coil sensitivity profiles. Moreover, it is a novel tool for reconstructing undersampled single coil k-space data. CONCLUSION Our approach avoids overfitting to dataset features, that can occur in Neural Networks trained on databases, because the network parameters are tuned only on the reconstruction data. It allows better results and faster reconstruction than the baseline untrained Neural Network approach.
Collapse
Affiliation(s)
- Thomas M Siedler
- Department of Experimental Physics 5, University of Würzburg, Würzburg, Germany
| | - Peter M Jakob
- Department of Experimental Physics 5, University of Würzburg, Würzburg, Germany
| | - Volker Herold
- Department of Experimental Physics 5, University of Würzburg, Würzburg, Germany
| |
Collapse
|
3
|
Wang S, Wu R, Jia S, Diakite A, Li C, Liu Q, Zheng H, Ying L. Knowledge-driven deep learning for fast MR imaging: Undersampled MR image reconstruction from supervised to un-supervised learning. Magn Reson Med 2024; 92:496-518. [PMID: 38624162 DOI: 10.1002/mrm.30105] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/03/2023] [Revised: 03/19/2024] [Accepted: 03/20/2024] [Indexed: 04/17/2024]
Abstract
Deep learning (DL) has emerged as a leading approach in accelerating MRI. It employs deep neural networks to extract knowledge from available datasets and then applies the trained networks to reconstruct accurate images from limited measurements. Unlike natural image restoration problems, MRI involves physics-based imaging processes, unique data properties, and diverse imaging tasks. This domain knowledge needs to be integrated with data-driven approaches. Our review will introduce the significant challenges faced by such knowledge-driven DL approaches in the context of fast MRI along with several notable solutions, which include learning neural networks and addressing different imaging application scenarios. The traits and trends of these techniques have also been given which have shifted from supervised learning to semi-supervised learning, and finally, to unsupervised learning methods. In addition, MR vendors' choices of DL reconstruction have been provided along with some discussions on open questions and future directions, which are critical for the reliable imaging systems.
Collapse
Affiliation(s)
- Shanshan Wang
- Paul C Lauterbur Research Center for Biomedical Imaging, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
| | - Ruoyou Wu
- Paul C Lauterbur Research Center for Biomedical Imaging, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
| | - Sen Jia
- Paul C Lauterbur Research Center for Biomedical Imaging, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
| | - Alou Diakite
- Paul C Lauterbur Research Center for Biomedical Imaging, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
- University of Chinese Academy of Sciences, Beijing, China
| | - Cheng Li
- Paul C Lauterbur Research Center for Biomedical Imaging, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
| | - Qiegen Liu
- Department of Electronic Information Engineering, Nanchang University, Nanchang, China
| | - Hairong Zheng
- Paul C Lauterbur Research Center for Biomedical Imaging, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
| | - Leslie Ying
- Department of Biomedical Engineering and Department of Electrical Engineering, The State University of New York, Buffalo, New York, USA
| |
Collapse
|
4
|
Liu X, Pang Y, Liu Y, Jin R, Sun Y, Liu Y, Xiao J. Dual-domain faster Fourier convolution based network for MR image reconstruction. Comput Biol Med 2024; 177:108603. [PMID: 38781646 DOI: 10.1016/j.compbiomed.2024.108603] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/31/2024] [Revised: 04/15/2024] [Accepted: 05/11/2024] [Indexed: 05/25/2024]
Abstract
Deep learning methods for fast MRI have shown promise in reconstructing high-quality images from undersampled multi-coil k-space data, leading to reduced scan duration. However, existing methods encounter challenges related to limited receptive fields in dual-domain (k-space and image domains) reconstruction networks, rigid data consistency operations, and suboptimal refinement structures, which collectively restrict overall reconstruction performance. This study introduces a comprehensive framework that addresses these challenges and enhances MR image reconstruction quality. Firstly, we propose Faster Inverse Fourier Convolution (FasterIFC), a frequency domain convolutional operator that significantly expands the receptive field of k-space domain reconstruction networks. Expanding the information extraction range to the entire frequency spectrum according to the spectral convolution theorem in Fourier theory enables the network to easily utilize richer redundant long-range information from adjacent, symmetrical, and diagonal locations of multi-coil k-space data. Secondly, we introduce a novel softer Data Consistency (softerDC) layer, which achieves an enhanced balance between data consistency and smoothness. This layer facilitates the implementation of diverse data consistency strategies across distinct frequency positions, addressing the inflexibility observed in current methods. Finally, we present the Dual-Domain Faster Fourier Convolution Based Network (D2F2), which features a centrosymmetric dual-domain parallel structure based on FasterIFC. This architecture optimally leverages dual-domain data characteristics while substantially expanding the receptive field in both domains. Coupled with the softerDC layer, D2F2 demonstrates superior performance on the NYU fastMRI dataset at multiple acceleration factors, surpassing state-of-the-art methods in both quantitative and qualitative evaluations.
Collapse
Affiliation(s)
- Xiaohan Liu
- TJK-BIIT Lab, School of Electrical and Information Engineering, Tianjin University, Tianjin, 300072, China; Tiandatz Technology Co. Ltd., Tianjin, 300072, China.
| | - Yanwei Pang
- TJK-BIIT Lab, School of Electrical and Information Engineering, Tianjin University, Tianjin, 300072, China.
| | - Yiming Liu
- TJK-BIIT Lab, School of Electrical and Information Engineering, Tianjin University, Tianjin, 300072, China.
| | - Ruiqi Jin
- TJK-BIIT Lab, School of Electrical and Information Engineering, Tianjin University, Tianjin, 300072, China.
| | - Yong Sun
- TJK-BIIT Lab, School of Electrical and Information Engineering, Tianjin University, Tianjin, 300072, China.
| | - Yu Liu
- TJK-BIIT Lab, School of Electrical and Information Engineering, Tianjin University, Tianjin, 300072, China.
| | - Jing Xiao
- TJK-BIIT Lab, School of Electrical and Information Engineering, Tianjin University, Tianjin, 300072, China; Department of Economic Management, Hebei Chemical and Pharmaceutical College, Shijiazhuang, Hebei, 050026, China.
| |
Collapse
|
5
|
Chaudhary MFA, Gerard SE, Christensen GE, Cooper CB, Schroeder JD, Hoffman EA, Reinhardt JM. LungViT: Ensembling Cascade of Texture Sensitive Hierarchical Vision Transformers for Cross-Volume Chest CT Image-to-Image Translation. IEEE TRANSACTIONS ON MEDICAL IMAGING 2024; 43:2448-2465. [PMID: 38373126 PMCID: PMC11227912 DOI: 10.1109/tmi.2024.3367321] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 02/21/2024]
Abstract
Chest computed tomography (CT) at inspiration is often complemented by an expiratory CT to identify peripheral airways disease. Additionally, co-registered inspiratory-expiratory volumes can be used to derive various markers of lung function. Expiratory CT scans, however, may not be acquired due to dose or scan time considerations or may be inadequate due to motion or insufficient exhale; leading to a missed opportunity to evaluate underlying small airways disease. Here, we propose LungViT- a generative adversarial learning approach using hierarchical vision transformers for translating inspiratory CT intensities to corresponding expiratory CT intensities. LungViT addresses several limitations of the traditional generative models including slicewise discontinuities, limited size of generated volumes, and their inability to model texture transfer at volumetric level. We propose a shifted-window hierarchical vision transformer architecture with squeeze-and-excitation decoder blocks for modeling dependencies between features. We also propose a multiview texture similarity distance metric for texture and style transfer in 3D. To incorporate global information into the training process and refine the output of our model, we use ensemble cascading. LungViT is able to generate large 3D volumes of size 320×320×320 . We train and validate our model using a diverse cohort of 1500 subjects with varying disease severity. To assess model generalizability beyond the development set biases, we evaluate our model on an out-of-distribution external validation set of 200 subjects. Clinical validation on internal and external testing sets shows that synthetic volumes could be reliably adopted for deriving clinical endpoints of chronic obstructive pulmonary disease.
Collapse
|
6
|
Zhao X, Yang T, Li B, Yang A, Yan Y, Jiao C. DiffGAN: An adversarial diffusion model with local transformer for MRI reconstruction. Magn Reson Imaging 2024; 109:108-119. [PMID: 38492787 DOI: 10.1016/j.mri.2024.03.017] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/16/2023] [Revised: 03/13/2024] [Accepted: 03/13/2024] [Indexed: 03/18/2024]
Abstract
Magnetic resonance imaging (MRI) is non-invasive and crucial for clinical diagnosis, but it has long acquisition time and aliasing artifacts. Accelerated imaging techniques can effectively reduce the scanning time of MRI, thereby decreasing the anxiety and discomfort of patients. Vision Transformer (ViT) based methods have greatly improved MRI image reconstruction, but their computational complexity and memory requirements for the self-attention mechanism grow quadratically with image resolution, which limits their use for high resolution images. In addition, the current generative adversarial networks in MRI reconstruction are difficult to train stably. To address these problems, we propose a Local Vision Transformer (LVT) based adversarial Diffusion model (Diff-GAN) for accelerating MRI reconstruction. We employ a generative adversarial network (GAN) as the reverse diffusion model to enable large diffusion steps. In the forward diffusion module, we use a diffusion process to generate Gaussian mixture distribution noise, which mitigates the gradient vanishing issue in GAN training. This network leverages the LVT module with the local self-attention, which can capture high-quality local features and detailed information. We evaluate our method on four datasets: IXI, MICCAI 2013, MRNet and FastMRI, and demonstrate that Diff-GAN can outperform several state-of-the-art GAN-based methods for MRI reconstruction.
Collapse
Affiliation(s)
- Xiang Zhao
- School of Information Science and Engineering, Henan University of Technology, Zhengzhou 450001, China
| | - Tiejun Yang
- School of Artificial Intelligence and Big Data, Henan University of Technology, Zhengzhou 450001, China; Key Laboratory of Grain Information Processing and Control (HAUT), Ministry of Education, Zhengzhou, China; Henan Key Laboratory of Grain Photoelectric Detection and Control (HAUT), Zhengzhou, Henan, China.
| | - Bingjie Li
- School of Information Science and Engineering, Henan University of Technology, Zhengzhou 450001, China
| | - Aolin Yang
- School of Information Science and Engineering, Henan University of Technology, Zhengzhou 450001, China
| | - Yanghui Yan
- School of Information Science and Engineering, Henan University of Technology, Zhengzhou 450001, China
| | - Chunxia Jiao
- School of Information Science and Engineering, Henan University of Technology, Zhengzhou 450001, China
| |
Collapse
|
7
|
Suwannasak A, Angkurawaranon S, Sangpin P, Chatnuntawech I, Wantanajittikul K, Yarach U. Deep learning-based super-resolution of structural brain MRI at 1.5 T: application to quantitative volume measurement. MAGMA (NEW YORK, N.Y.) 2024:10.1007/s10334-024-01165-8. [PMID: 38758489 DOI: 10.1007/s10334-024-01165-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/13/2023] [Revised: 04/27/2024] [Accepted: 04/30/2024] [Indexed: 05/18/2024]
Abstract
OBJECTIVE This study investigated the feasibility of using deep learning-based super-resolution (DL-SR) technique on low-resolution (LR) images to generate high-resolution (HR) MR images with the aim of scan time reduction. The efficacy of DL-SR was also assessed through the application of brain volume measurement (BVM). MATERIALS AND METHODS In vivo brain images acquired with 3D-T1W from various MRI scanners were utilized. For model training, LR images were generated by downsampling the original 1 mm-2 mm isotropic resolution images. Pairs of LR and HR images were used for training 3D residual dense net (RDN). For model testing, actual scanned 2 mm isotropic resolution 3D-T1W images with one-minute scan time were used. Normalized root-mean-square error (NRMSE), peak signal-to-noise ratio (PSNR), and structural similarity (SSIM) were used for model evaluation. The evaluation also included brain volume measurement, with assessments of subcortical brain regions. RESULTS The results showed that DL-SR model improved the quality of LR images compared with cubic interpolation, as indicated by NRMSE (24.22% vs 30.13%), PSNR (26.19 vs 24.65), and SSIM (0.96 vs 0.95). For volumetric assessments, there were no significant differences between DL-SR and actual HR images (p > 0.05, Pearson's correlation > 0.90) at seven subcortical regions. DISCUSSION The combination of LR MRI and DL-SR enables addressing prolonged scan time in 3D MRI scans while providing sufficient image quality without affecting brain volume measurement.
Collapse
Affiliation(s)
- Atita Suwannasak
- Department of Radiologic Technology, Faculty of Associated Medical Sciences, Chiang Mai University, 110 Intavaroros Road, Muang, Chiang Mai, 50200, Thailand
| | - Salita Angkurawaranon
- Department of Radiology, Faculty of Medicine, Chiang Mai University, Intavaroros Road, Muang, Chiang Mai, Thailand
| | - Prapatsorn Sangpin
- Philips (Thailand) Ltd, New Petchburi Road, Bangkapi, Huaykwang, Bangkok, Thailand
| | - Itthi Chatnuntawech
- National Nanotechnology Center (NANOTEC), Phahon Yothin Road, Khlong Nueng, Khlong Luang, Pathum Thani, Thailand
| | - Kittichai Wantanajittikul
- Department of Radiologic Technology, Faculty of Associated Medical Sciences, Chiang Mai University, 110 Intavaroros Road, Muang, Chiang Mai, 50200, Thailand
| | - Uten Yarach
- Department of Radiologic Technology, Faculty of Associated Medical Sciences, Chiang Mai University, 110 Intavaroros Road, Muang, Chiang Mai, 50200, Thailand.
| |
Collapse
|
8
|
Chen S, Duan J, Ren X, Wang J, Liu Y. DFUSNN: zero-shot dual-domain fusion unsupervised neural network for parallel MRI reconstruction. Phys Med Biol 2024; 69:105028. [PMID: 38604186 DOI: 10.1088/1361-6560/ad3dbc] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/23/2023] [Accepted: 04/11/2024] [Indexed: 04/13/2024]
Abstract
Objective. Recently, deep learning models have been used to reconstruct parallel magnetic resonance (MR) images from undersampled k-space data. However, most existing approaches depend on large databases of fully sampled MR data for training, which can be challenging or sometimes infeasible to acquire in certain scenarios. The goal is to develop an effective alternative for improved reconstruction quality that does not rely on external training datasets.Approach. We introduce a novel zero-shot dual-domain fusion unsupervised neural network (DFUSNN) for parallel MR imaging reconstruction without any external training datasets. We employ the Noise2Noise (N2N) network for the reconstruction in the k-space domain, integrate phase and coil sensitivity smoothness priors into the k-space N2N network, and use an early stopping criterion to prevent overfitting. Additionally, we propose a dual-domain fusion method based on Bayesian optimization to enhance reconstruction quality efficiently.Results. Simulation experiments conducted on three datasets with different undersampling patterns showed that the DFUSNN outperforms all other competing unsupervised methods and the one-shot Hankel-k-space generative model (HKGM). The DFUSNN also achieves comparable results to the supervised Deep-SLR method.Significance. The novel DFUSNN model offers a viable solution for reconstructing high-quality MR images without the need for external training datasets, thereby overcoming a major hurdle in scenarios where acquiring fully sampled MR data is difficult.
Collapse
Affiliation(s)
- Shengyi Chen
- Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Kunming 650500, People's Republic of China
| | - Jizhong Duan
- Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Kunming 650500, People's Republic of China
| | - Xinmin Ren
- Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Kunming 650500, People's Republic of China
| | - Junfeng Wang
- Department of Hepatobiliary Surgery, First People's Hospital of Yunnan Province, Kunming 650030, People's Republic of China
| | - Yu Liu
- School of Microelectronics, Tianjin University, Tianjin 300072, People's Republic of China
| |
Collapse
|
9
|
Hossain MB, Shinde RK, Imtiaz SM, Hossain FMF, Jeon SH, Kwon KC, Kim N. Swin Transformer and the Unet Architecture to Correct Motion Artifacts in Magnetic Resonance Image Reconstruction. Int J Biomed Imaging 2024; 2024:8972980. [PMID: 38725808 PMCID: PMC11081754 DOI: 10.1155/2024/8972980] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/14/2023] [Revised: 04/08/2024] [Accepted: 04/23/2024] [Indexed: 05/12/2024] Open
Abstract
We present a deep learning-based method that corrects motion artifacts and thus accelerates data acquisition and reconstruction of magnetic resonance images. The novel model, the Motion Artifact Correction by Swin Network (MACS-Net), uses a Swin transformer layer as the fundamental block and the Unet architecture as the neural network backbone. We employ a hierarchical transformer with shifted windows to extract multiscale contextual features during encoding. A new dual upsampling technique is employed to enhance the spatial resolutions of feature maps in the Swin transformer-based decoder layer. A raw magnetic resonance imaging dataset is used for network training and testing; the data contain various motion artifacts with ground truth images of the same subjects. The results were compared to six state-of-the-art MRI image motion correction methods using two types of motions. When motions were brief (within 5 s), the method reduced the average normalized root mean square error (NRMSE) from 45.25% to 17.51%, increased the mean structural similarity index measure (SSIM) from 79.43% to 91.72%, and increased the peak signal-to-noise ratio (PSNR) from 18.24 to 26.57 dB. Similarly, when motions were extended from 5 to 10 s, our approach decreased the average NRMSE from 60.30% to 21.04%, improved the mean SSIM from 33.86% to 90.33%, and increased the PSNR from 15.64 to 24.99 dB. The anatomical structures of the corrected images and the motion-free brain data were similar.
Collapse
Affiliation(s)
- Md. Biddut Hossain
- Department of Information and Communication Engineering, Chungbuk National University, Cheongju-si 28644, Chungcheongbuk-do, Republic of Korea
| | - Rupali Kiran Shinde
- Department of Information and Communication Engineering, Chungbuk National University, Cheongju-si 28644, Chungcheongbuk-do, Republic of Korea
| | - Shariar Md Imtiaz
- Department of Information and Communication Engineering, Chungbuk National University, Cheongju-si 28644, Chungcheongbuk-do, Republic of Korea
| | - F. M. Fahmid Hossain
- Department of Information and Communication Engineering, Chungbuk National University, Cheongju-si 28644, Chungcheongbuk-do, Republic of Korea
| | - Seok-Hee Jeon
- Department of Electronics Engineering, Incheon National University, 119 Academy-ro, Yeonsu-gu, Incheon 22012, Republic of Korea
| | - Ki-Chul Kwon
- Department of Information and Communication Engineering, Chungbuk National University, Cheongju-si 28644, Chungcheongbuk-do, Republic of Korea
| | - Nam Kim
- Department of Information and Communication Engineering, Chungbuk National University, Cheongju-si 28644, Chungcheongbuk-do, Republic of Korea
| |
Collapse
|
10
|
Chen Z, Niu C, Gao Q, Wang G, Shan H. LIT-Former: Linking In-Plane and Through-Plane Transformers for Simultaneous CT Image Denoising and Deblurring. IEEE TRANSACTIONS ON MEDICAL IMAGING 2024; 43:1880-1894. [PMID: 38194396 DOI: 10.1109/tmi.2024.3351723] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 01/11/2024]
Abstract
This paper studies 3D low-dose computed tomography (CT) imaging. Although various deep learning methods were developed in this context, typically they focus on 2D images and perform denoising due to low-dose and deblurring for super-resolution separately. Up to date, little work was done for simultaneous in-plane denoising and through-plane deblurring, which is important to obtain high-quality 3D CT images with lower radiation and faster imaging speed. For this task, a straightforward method is to directly train an end-to-end 3D network. However, it demands much more training data and expensive computational costs. Here, we propose to link in-plane and through-plane transformers for simultaneous in-plane denoising and through-plane deblurring, termed as LIT-Former, which can efficiently synergize in-plane and through-plane sub-tasks for 3D CT imaging and enjoy the advantages of both convolution and transformer networks. LIT-Former has two novel designs: efficient multi-head self-attention modules (eMSM) and efficient convolutional feed-forward networks (eCFN). First, eMSM integrates in-plane 2D self-attention and through-plane 1D self-attention to efficiently capture global interactions of 3D self-attention, the core unit of transformer networks. Second, eCFN integrates 2D convolution and 1D convolution to extract local information of 3D convolution in the same fashion. As a result, the proposed LIT-Former synergizes these two sub-tasks, significantly reducing the computational complexity as compared to 3D counterparts and enabling rapid convergence. Extensive experimental results on simulated and clinical datasets demonstrate superior performance over state-of-the-art models. The source code is made available at https://github.com/hao1635/LIT-Former.
Collapse
|
11
|
Dalmaz O, Mirza MU, Elmas G, Ozbey M, Dar SUH, Ceyani E, Oguz KK, Avestimehr S, Çukur T. One model to unite them all: Personalized federated learning of multi-contrast MRI synthesis. Med Image Anal 2024; 94:103121. [PMID: 38402791 DOI: 10.1016/j.media.2024.103121] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/26/2023] [Revised: 02/20/2024] [Accepted: 02/21/2024] [Indexed: 02/27/2024]
Abstract
Curation of large, diverse MRI datasets via multi-institutional collaborations can help improve learning of generalizable synthesis models that reliably translate source- onto target-contrast images. To facilitate collaborations, federated learning (FL) adopts decentralized model training while mitigating privacy concerns by avoiding sharing of imaging data. However, conventional FL methods can be impaired by the inherent heterogeneity in the data distribution, with domain shifts evident within and across imaging sites. Here we introduce the first personalized FL method for MRI Synthesis (pFLSynth) that improves reliability against data heterogeneity via model specialization to individual sites and synthesis tasks (i.e., source-target contrasts). To do this, pFLSynth leverages an adversarial model equipped with novel personalization blocks that control the statistics of generated feature maps across the spatial/channel dimensions, given latent variables specific to sites and tasks. To further promote communication efficiency and site specialization, partial network aggregation is employed over later generator stages while earlier generator stages and the discriminator are trained locally. As such, pFLSynth enables multi-task training of multi-site synthesis models with high generalization performance across sites and tasks. Comprehensive experiments demonstrate the superior performance and reliability of pFLSynth in MRI synthesis against prior federated methods.
Collapse
Affiliation(s)
- Onat Dalmaz
- Department of Electrical and Electronics Engineering, Bilkent University, Ankara 06800, Turkey; National Magnetic Resonance Research Center (UMRAM), Bilkent University, Ankara 06800, Turkey
| | - Muhammad U Mirza
- Department of Electrical and Electronics Engineering, Bilkent University, Ankara 06800, Turkey; National Magnetic Resonance Research Center (UMRAM), Bilkent University, Ankara 06800, Turkey
| | - Gokberk Elmas
- Department of Electrical and Electronics Engineering, Bilkent University, Ankara 06800, Turkey; National Magnetic Resonance Research Center (UMRAM), Bilkent University, Ankara 06800, Turkey
| | - Muzaffer Ozbey
- Department of Electrical and Electronics Engineering, Bilkent University, Ankara 06800, Turkey; National Magnetic Resonance Research Center (UMRAM), Bilkent University, Ankara 06800, Turkey
| | - Salman U H Dar
- Department of Electrical and Electronics Engineering, Bilkent University, Ankara 06800, Turkey; National Magnetic Resonance Research Center (UMRAM), Bilkent University, Ankara 06800, Turkey
| | - Emir Ceyani
- Department of Electrical and Computer Engineering, University of Southern California, Los Angeles, CA 90089, USA
| | - Kader K Oguz
- Department of Radiology, University of California, Davis Medical Center, Sacramento, CA 95817, USA
| | - Salman Avestimehr
- Department of Electrical and Computer Engineering, University of Southern California, Los Angeles, CA 90089, USA
| | - Tolga Çukur
- Department of Electrical and Electronics Engineering, Bilkent University, Ankara 06800, Turkey; National Magnetic Resonance Research Center (UMRAM), Bilkent University, Ankara 06800, Turkey; Neuroscience Program, Bilkent University, Ankara 06800, Turkey.
| |
Collapse
|
12
|
Yan Y, Yang T, Jiao C, Yang A, Miao J. IWNeXt: an image-wavelet domain ConvNeXt-based network for self-supervised multi-contrast MRI reconstruction. Phys Med Biol 2024; 69:085005. [PMID: 38479022 DOI: 10.1088/1361-6560/ad33b4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/08/2023] [Accepted: 03/13/2024] [Indexed: 04/04/2024]
Abstract
Objective.Multi-contrast magnetic resonance imaging (MC MRI) can obtain more comprehensive anatomical information of the same scanning object but requires a longer acquisition time than single-contrast MRI. To accelerate MC MRI speed, recent studies only collect partial k-space data of one modality (target contrast) to reconstruct the remaining non-sampled measurements using a deep learning-based model with the assistance of another fully sampled modality (reference contrast). However, MC MRI reconstruction mainly performs the image domain reconstruction with conventional CNN-based structures by full supervision. It ignores the prior information from reference contrast images in other sparse domains and requires fully sampled target contrast data. In addition, because of the limited receptive field, conventional CNN-based networks are difficult to build a high-quality non-local dependency.Approach.In the paper, we propose an Image-Wavelet domain ConvNeXt-based network (IWNeXt) for self-supervised MC MRI reconstruction. Firstly, INeXt and WNeXt based on ConvNeXt reconstruct undersampled target contrast data in the image domain and refine the initial reconstructed result in the wavelet domain respectively. To generate more tissue details in the refinement stage, reference contrast wavelet sub-bands are used as additional supplementary information for wavelet domain reconstruction. Then we design a novel attention ConvNeXt block for feature extraction, which can capture the non-local information of the MC image. Finally, the cross-domain consistency loss is designed for self-supervised learning. Especially, the frequency domain consistency loss deduces the non-sampled data, while the image and wavelet domain consistency loss retain more high-frequency information in the final reconstruction.Main results.Numerous experiments are conducted on the HCP dataset and the M4Raw dataset with different sampling trajectories. Compared with DuDoRNet, our model improves by 1.651 dB in the peak signal-to-noise ratio.Significance.IWNeXt is a potential cross-domain method that can enhance the accuracy of MC MRI reconstruction and reduce reliance on fully sampled target contrast images.
Collapse
Affiliation(s)
- Yanghui Yan
- School of Information Science and Engineering, Henan University of Technology, Zhengzhou 450001, People's Republic of China
| | - Tiejun Yang
- School of Artificial Intelligence and Big Data, Henan University of Technology, Zhengzhou, 450001, People's Republic of China
- Key Laboratory of Grain Information Processing and Control (HAUT), Ministry of Education, Zhengzhou, People's Republic of China
- Henan Key Laboratory of Grain Photoelectric Detection and Control (HAUT), Zhengzhou, Henan, People's Republic of China
| | - Chunxia Jiao
- School of Information Science and Engineering, Henan University of Technology, Zhengzhou 450001, People's Republic of China
| | - Aolin Yang
- School of Information Science and Engineering, Henan University of Technology, Zhengzhou 450001, People's Republic of China
| | - Jianyu Miao
- School of Artificial Intelligence and Big Data, Henan University of Technology, Zhengzhou, 450001, People's Republic of China
| |
Collapse
|
13
|
Bao Q, Liu X, Xu J, Xia L, Otikovs M, Xie H, Liu K, Zhang Z, Zhou X, Liu C. Unsupervised deep learning model for correcting Nyquist ghosts of single-shot spatiotemporal encoding. Magn Reson Med 2024; 91:1368-1383. [PMID: 38073072 DOI: 10.1002/mrm.29925] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/15/2023] [Revised: 10/20/2023] [Accepted: 10/25/2023] [Indexed: 02/03/2024]
Abstract
PURPOSE To design an unsupervised deep learning (DL) model for correcting Nyquist ghosts of single-shot spatiotemporal encoding (SPEN) and evaluate the model for real MRI applications. METHODS The proposed method consists of three main components: (1) an unsupervised network that combines Residual Encoder and Restricted Subspace Mapping (RERSM-net) and is trained to generate a phase-difference map based on the even and odd SPEN images; (2) a spin physical forward model to obtain the corrected image with the learned phase difference map; and (3) cycle-consistency loss that is explored for training the RERSM-net. RESULTS The proposed RERSM-net could effectively generate smooth phase difference maps and correct Nyquist ghosts of single-shot SPEN. Both simulation and real in vivo MRI experiments demonstrated that our method outperforms the state-of-the-art SPEN Nyquist ghost correction method. Furthermore, the ablation experiments of generating phase-difference maps show the advantages of the proposed unsupervised model. CONCLUSION The proposed method can effectively correct Nyquist ghosts for the single-shot SPEN sequence.
Collapse
Affiliation(s)
- Qingjia Bao
- Key Laboratory of Magnetic Resonance in Biological Systems, Innovation Academy for Precision Measurement Science and Technology, Wuhan, China
| | - Xinjie Liu
- Key Laboratory of Magnetic Resonance in Biological Systems, Innovation Academy for Precision Measurement Science and Technology, Wuhan, China
- University of Chinese Academy of Sciences, Beijing, China
| | - Jingyun Xu
- School of Information Engineering, Wuhan University of Technology, Wuhan, China
| | - Liyang Xia
- School of Information Engineering, Wuhan University of Technology, Wuhan, China
| | | | - Han Xie
- Key Laboratory of Magnetic Resonance in Biological Systems, Innovation Academy for Precision Measurement Science and Technology, Wuhan, China
| | - Kewen Liu
- School of Information Engineering, Wuhan University of Technology, Wuhan, China
| | - Zhi Zhang
- Key Laboratory of Magnetic Resonance in Biological Systems, Innovation Academy for Precision Measurement Science and Technology, Wuhan, China
| | - Xin Zhou
- Key Laboratory of Magnetic Resonance in Biological Systems, Innovation Academy for Precision Measurement Science and Technology, Wuhan, China
- University of Chinese Academy of Sciences, Beijing, China
- Optics Valley Laboratory, Wuhan, China
| | - Chaoyang Liu
- Key Laboratory of Magnetic Resonance in Biological Systems, Innovation Academy for Precision Measurement Science and Technology, Wuhan, China
- University of Chinese Academy of Sciences, Beijing, China
- Optics Valley Laboratory, Wuhan, China
| |
Collapse
|
14
|
Liu Y, Zhang Z, Yue J, Guo W. SCANeXt: Enhancing 3D medical image segmentation with dual attention network and depth-wise convolution. Heliyon 2024; 10:e26775. [PMID: 38439873 PMCID: PMC10909707 DOI: 10.1016/j.heliyon.2024.e26775] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/05/2024] [Revised: 02/19/2024] [Accepted: 02/20/2024] [Indexed: 03/06/2024] Open
Abstract
Existing approaches to 3D medical image segmentation can be generally categorized into convolution-based or transformer-based methods. While convolutional neural networks (CNNs) demonstrate proficiency in extracting local features, they encounter challenges in capturing global representations. In contrast, the consecutive self-attention modules present in vision transformers excel at capturing long-range dependencies and achieving an expanded receptive field. In this paper, we propose a novel approach, termed SCANeXt, for 3D medical image segmentation. Our method combines the strengths of dual attention (Spatial and Channel Attention) and ConvNeXt to enhance representation learning for 3D medical images. In particular, we propose a novel self-attention mechanism crafted to encompass spatial and channel relationships throughout the entire feature dimension. To further extract multiscale features, we introduce a depth-wise convolution block inspired by ConvNeXt after the dual attention block. Extensive evaluations on three benchmark datasets, namely Synapse, BraTS, and ACDC, demonstrate the effectiveness of our proposed method in terms of accuracy. Our SCANeXt model achieves a state-of-the-art result with a Dice Similarity Score of 95.18% on the ACDC dataset, significantly outperforming current methods.
Collapse
Affiliation(s)
- Yajun Liu
- Shanghai Key Laboratory of Intelligent Sensing and Recognition, Shanghai Jiao Tong University, China
| | - Zenghui Zhang
- Shanghai Key Laboratory of Intelligent Sensing and Recognition, Shanghai Jiao Tong University, China
| | - Jiang Yue
- Department of Endocrinology and Metabolism, Renji Hospital, School of Medicine, Shanghai Jiao Tong University, China
| | - Weiwei Guo
- Center for Digital Innovation, Tongji University, China
| |
Collapse
|
15
|
Huang J, Ferreira PF, Wang L, Wu Y, Aviles-Rivero AI, Schönlieb CB, Scott AD, Khalique Z, Dwornik M, Rajakulasingam R, De Silva R, Pennell DJ, Nielles-Vallespin S, Yang G. Deep learning-based diffusion tensor cardiac magnetic resonance reconstruction: a comparison study. Sci Rep 2024; 14:5658. [PMID: 38454072 PMCID: PMC10920645 DOI: 10.1038/s41598-024-55880-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/05/2023] [Accepted: 02/27/2024] [Indexed: 03/09/2024] Open
Abstract
In vivo cardiac diffusion tensor imaging (cDTI) is a promising Magnetic Resonance Imaging (MRI) technique for evaluating the microstructure of myocardial tissue in living hearts, providing insights into cardiac function and enabling the development of innovative therapeutic strategies. However, the integration of cDTI into routine clinical practice poses challenging due to the technical obstacles involved in the acquisition, such as low signal-to-noise ratio and prolonged scanning times. In this study, we investigated and implemented three different types of deep learning-based MRI reconstruction models for cDTI reconstruction. We evaluated the performance of these models based on the reconstruction quality assessment, the diffusion tensor parameter assessment as well as the computational cost assessment. Our results indicate that the models discussed in this study can be applied for clinical use at an acceleration factor (AF) of × 2 and × 4 , with the D5C5 model showing superior fidelity for reconstruction and the SwinMR model providing higher perceptual scores. There is no statistical difference from the reference for all diffusion tensor parameters at AF × 2 or most DT parameters at AF × 4 , and the quality of most diffusion tensor parameter maps is visually acceptable. SwinMR is recommended as the optimal approach for reconstruction at AF × 2 and AF × 4 . However, we believe that the models discussed in this study are not yet ready for clinical use at a higher AF. At AF × 8 , the performance of all models discussed remains limited, with only half of the diffusion tensor parameters being recovered to a level with no statistical difference from the reference. Some diffusion tensor parameter maps even provide wrong and misleading information.
Collapse
Grants
- Wellcome Trust
- RG/19/1/34160 British Heart Foundation
- This study was supported in part by the UKRI Future Leaders Fellowship (MR/V023799/1), BHF (RG/19/1/34160), the ERC IMI (101005122), the H2020 (952172), the MRC (MC/PC/21013), the Royal Society (IEC/NSFC/211235), the NVIDIA Academic Hardware Grant Program, EPSRC (EP/V029428/1, EP/S026045/1, EP/T003553/1, EP/N014588/1, EP/T017961/1), and the Cambridge Mathematics of Information in Healthcare Hub (CMIH) Partnership Fund.
Collapse
Affiliation(s)
- Jiahao Huang
- National Heart and Lung Institute, Imperial College London, London, SW7 2AZ, UK.
- Cardiovascular Research Centre, Royal Brompton Hospital, London, SW7 2AZ, UK.
- Bioengineering Department and Imperial-X, Imperial College London, London, W12 7SL, UK.
| | - Pedro F Ferreira
- National Heart and Lung Institute, Imperial College London, London, SW7 2AZ, UK
- Cardiovascular Research Centre, Royal Brompton Hospital, London, SW7 2AZ, UK
| | - Lichao Wang
- National Heart and Lung Institute, Imperial College London, London, SW7 2AZ, UK
- Department of Computing, Imperial College London, London, UK
| | - Yinzhe Wu
- National Heart and Lung Institute, Imperial College London, London, SW7 2AZ, UK
- Cardiovascular Research Centre, Royal Brompton Hospital, London, SW7 2AZ, UK
| | - Angelica I Aviles-Rivero
- Department of Applied Mathematics and Theoretical Physics, University of Cambridge, Cambridge, UK
| | - Carola-Bibiane Schönlieb
- Department of Applied Mathematics and Theoretical Physics, University of Cambridge, Cambridge, UK
| | - Andrew D Scott
- National Heart and Lung Institute, Imperial College London, London, SW7 2AZ, UK
- Cardiovascular Research Centre, Royal Brompton Hospital, London, SW7 2AZ, UK
| | - Zohya Khalique
- National Heart and Lung Institute, Imperial College London, London, SW7 2AZ, UK
- Cardiovascular Research Centre, Royal Brompton Hospital, London, SW7 2AZ, UK
| | - Maria Dwornik
- National Heart and Lung Institute, Imperial College London, London, SW7 2AZ, UK
- Cardiovascular Research Centre, Royal Brompton Hospital, London, SW7 2AZ, UK
| | - Ramyah Rajakulasingam
- National Heart and Lung Institute, Imperial College London, London, SW7 2AZ, UK
- Cardiovascular Research Centre, Royal Brompton Hospital, London, SW7 2AZ, UK
| | - Ranil De Silva
- National Heart and Lung Institute, Imperial College London, London, SW7 2AZ, UK
- Cardiovascular Research Centre, Royal Brompton Hospital, London, SW7 2AZ, UK
| | - Dudley J Pennell
- National Heart and Lung Institute, Imperial College London, London, SW7 2AZ, UK
- Cardiovascular Research Centre, Royal Brompton Hospital, London, SW7 2AZ, UK
| | - Sonia Nielles-Vallespin
- National Heart and Lung Institute, Imperial College London, London, SW7 2AZ, UK
- Cardiovascular Research Centre, Royal Brompton Hospital, London, SW7 2AZ, UK
| | - Guang Yang
- National Heart and Lung Institute, Imperial College London, London, SW7 2AZ, UK.
- Cardiovascular Research Centre, Royal Brompton Hospital, London, SW7 2AZ, UK.
- Bioengineering Department and Imperial-X, Imperial College London, London, W12 7SL, UK.
| |
Collapse
|
16
|
Hossain MB, Shinde RK, Oh S, Kwon KC, Kim N. A Systematic Review and Identification of the Challenges of Deep Learning Techniques for Undersampled Magnetic Resonance Image Reconstruction. SENSORS (BASEL, SWITZERLAND) 2024; 24:753. [PMID: 38339469 PMCID: PMC10856856 DOI: 10.3390/s24030753] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/30/2023] [Revised: 01/05/2024] [Accepted: 01/22/2024] [Indexed: 02/12/2024]
Abstract
Deep learning (DL) in magnetic resonance imaging (MRI) shows excellent performance in image reconstruction from undersampled k-space data. Artifact-free and high-quality MRI reconstruction is essential for ensuring accurate diagnosis, supporting clinical decision-making, enhancing patient safety, facilitating efficient workflows, and contributing to the validity of research studies and clinical trials. Recently, deep learning has demonstrated several advantages over conventional MRI reconstruction methods. Conventional methods rely on manual feature engineering to capture complex patterns and are usually computationally demanding due to their iterative nature. Conversely, DL methods use neural networks with hundreds of thousands of parameters and automatically learn relevant features and representations directly from the data. Nevertheless, there are some limitations to DL-based techniques concerning MRI reconstruction tasks, such as the need for large, labeled datasets, the possibility of overfitting, and the complexity of model training. Researchers are striving to develop DL models that are more efficient, adaptable, and capable of providing valuable information for medical practitioners. We provide a comprehensive overview of the current developments and clinical uses by focusing on state-of-the-art DL architectures and tools used in MRI reconstruction. This study has three objectives. Our main objective is to describe how various DL designs have changed over time and talk about cutting-edge tactics, including their advantages and disadvantages. Hence, data pre- and post-processing approaches are assessed using publicly available MRI datasets and source codes. Secondly, this work aims to provide an extensive overview of the ongoing research on transformers and deep convolutional neural networks for rapid MRI reconstruction. Thirdly, we discuss several network training strategies, like supervised, unsupervised, transfer learning, and federated learning for rapid and efficient MRI reconstruction. Consequently, this article provides significant resources for future improvement of MRI data pre-processing and fast image reconstruction.
Collapse
Affiliation(s)
- Md. Biddut Hossain
- School of Information and Communication Engineering, Chungbuk National University, Cheongju-si 28644, Chungcheongbuk-do, Republic of Korea; (M.B.H.); (R.K.S.)
| | - Rupali Kiran Shinde
- School of Information and Communication Engineering, Chungbuk National University, Cheongju-si 28644, Chungcheongbuk-do, Republic of Korea; (M.B.H.); (R.K.S.)
| | - Sukhoon Oh
- Research Equipment Operation Department, Korea Basic Science Institute, Cheongju-si 28119, Chungcheongbuk-do, Republic of Korea;
| | - Ki-Chul Kwon
- School of Information and Communication Engineering, Chungbuk National University, Cheongju-si 28644, Chungcheongbuk-do, Republic of Korea; (M.B.H.); (R.K.S.)
| | - Nam Kim
- School of Information and Communication Engineering, Chungbuk National University, Cheongju-si 28644, Chungcheongbuk-do, Republic of Korea; (M.B.H.); (R.K.S.)
| |
Collapse
|
17
|
Ekanayake M, Pawar K, Harandi M, Egan G, Chen Z. McSTRA: A multi-branch cascaded swin transformer for point spread function-guided robust MRI reconstruction. Comput Biol Med 2024; 168:107775. [PMID: 38061154 DOI: 10.1016/j.compbiomed.2023.107775] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/01/2023] [Revised: 11/23/2023] [Accepted: 11/27/2023] [Indexed: 01/10/2024]
Abstract
Deep learning MRI reconstruction methods are often based on Convolutional neural network (CNN) models; however, they are limited in capturing global correlations among image features due to the intrinsic locality of the convolution operation. Conversely, the recent vision transformer models (ViT) are capable of capturing global correlations by applying self-attention operations on image patches. Nevertheless, the existing transformer models for MRI reconstruction rarely leverage the physics of MRI. In this paper, we propose a novel physics-based transformer model titled, the Multi-branch Cascaded Swin Transformers (McSTRA) for robust MRI reconstruction. McSTRA combines several interconnected MRI physics-related concepts with the Swin transformers: it exploits global MRI features via the shifted window self-attention mechanism; it extracts MRI features belonging to different spectral components via a multi-branch setup; it iterates between intermediate de-aliasing and data consistency via a cascaded network with intermediate loss computations; furthermore, we propose a point spread function-guided positional embedding generation mechanism for the Swin transformers which exploit the spread of the aliasing artifacts for effective reconstruction. With the combination of all these components, McSTRA outperforms the state-of-the-art methods while demonstrating robustness in adversarial conditions such as higher accelerations, noisy data, different undersampling protocols, out-of-distribution data, and abnormalities in anatomy.
Collapse
Affiliation(s)
- Mevan Ekanayake
- Monash Biomedical Imaging, Monash University, Australia; Department of Electrical and Computer Systems Engineering, Monash University, Australia.
| | - Kamlesh Pawar
- Monash Biomedical Imaging, Monash University, Australia
| | - Mehrtash Harandi
- Department of Electrical and Computer Systems Engineering, Monash University, Australia
| | - Gary Egan
- Monash Biomedical Imaging, Monash University, Australia; School of Psychological Sciences, Monash University, Australia
| | - Zhaolin Chen
- Monash Biomedical Imaging, Monash University, Australia; Department of Data Science and AI, Monash University, Australia
| |
Collapse
|
18
|
Zaid Alkilani A, Çukur T, Saritas EU. FD-Net: An unsupervised deep forward-distortion model for susceptibility artifact correction in EPI. Magn Reson Med 2024; 91:280-296. [PMID: 37811681 DOI: 10.1002/mrm.29851] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/10/2023] [Revised: 07/18/2023] [Accepted: 08/15/2023] [Indexed: 10/10/2023]
Abstract
PURPOSE To introduce an unsupervised deep-learning method for fast and effective correction of susceptibility artifacts in reversed phase-encode (PE) image pairs acquired with echo planar imaging (EPI). METHODS Recent learning-based correction approaches in EPI estimate a displacement field, unwarp the reversed-PE image pair with the estimated field, and average the unwarped pair to yield a corrected image. Unsupervised learning in these unwarping-based methods is commonly attained via a similarity constraint between the unwarped images in reversed-PE directions, neglecting consistency to the acquired EPI images. This work introduces a novel unsupervised deep Forward-Distortion Network (FD-Net) that predicts both the susceptibility-induced displacement field and the underlying anatomically correct image. Unlike previous methods, FD-Net enforces the forward-distortions of the correct image in both PE directions to be consistent with the acquired reversed-PE image pair. FD-Net further leverages a multiresolution architecture to maintain high local and global performance. RESULTS FD-Net performs competitively with a gold-standard reference method (TOPUP) in image quality, while enabling a leap in computational efficiency. Furthermore, FD-Net outperforms recent unwarping-based methods for unsupervised correction in terms of both image and field quality. CONCLUSION The unsupervised FD-Net method introduces a deep forward-distortion approach to enable fast, high-fidelity correction of susceptibility artifacts in EPI by maintaining consistency to measured data. Therefore, it holds great promise for improving the anatomical accuracy of EPI imaging.
Collapse
Affiliation(s)
- Abdallah Zaid Alkilani
- Department of Electrical and Electronics Engineering, Bilkent University, Ankara, Turkey
- National Magnetic Resonance Research Center (UMRAM), Bilkent University, Ankara, Turkey
| | - Tolga Çukur
- Department of Electrical and Electronics Engineering, Bilkent University, Ankara, Turkey
- National Magnetic Resonance Research Center (UMRAM), Bilkent University, Ankara, Turkey
- Neuroscience Graduate Program, Bilkent University, Ankara, Turkey
| | - Emine Ulku Saritas
- Department of Electrical and Electronics Engineering, Bilkent University, Ankara, Turkey
- National Magnetic Resonance Research Center (UMRAM), Bilkent University, Ankara, Turkey
- Neuroscience Graduate Program, Bilkent University, Ankara, Turkey
| |
Collapse
|
19
|
Azad R, Kazerouni A, Heidari M, Aghdam EK, Molaei A, Jia Y, Jose A, Roy R, Merhof D. Advances in medical image analysis with vision Transformers: A comprehensive review. Med Image Anal 2024; 91:103000. [PMID: 37883822 DOI: 10.1016/j.media.2023.103000] [Citation(s) in RCA: 6] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/09/2023] [Revised: 09/30/2023] [Accepted: 10/11/2023] [Indexed: 10/28/2023]
Abstract
The remarkable performance of the Transformer architecture in natural language processing has recently also triggered broad interest in Computer Vision. Among other merits, Transformers are witnessed as capable of learning long-range dependencies and spatial correlations, which is a clear advantage over convolutional neural networks (CNNs), which have been the de facto standard in Computer Vision problems so far. Thus, Transformers have become an integral part of modern medical image analysis. In this review, we provide an encyclopedic review of the applications of Transformers in medical imaging. Specifically, we present a systematic and thorough review of relevant recent Transformer literature for different medical image analysis tasks, including classification, segmentation, detection, registration, synthesis, and clinical report generation. For each of these applications, we investigate the novelty, strengths and weaknesses of the different proposed strategies and develop taxonomies highlighting key properties and contributions. Further, if applicable, we outline current benchmarks on different datasets. Finally, we summarize key challenges and discuss different future research directions. In addition, we have provided cited papers with their corresponding implementations in https://github.com/mindflow-institue/Awesome-Transformer.
Collapse
Affiliation(s)
- Reza Azad
- Faculty of Electrical Engineering and Information Technology, RWTH Aachen University, Aachen, Germany
| | - Amirhossein Kazerouni
- School of Electrical Engineering, Iran University of Science and Technology, Tehran, Iran
| | - Moein Heidari
- School of Electrical Engineering, Iran University of Science and Technology, Tehran, Iran
| | | | - Amirali Molaei
- School of Computer Engineering, Iran University of Science and Technology, Tehran, Iran
| | - Yiwei Jia
- Faculty of Electrical Engineering and Information Technology, RWTH Aachen University, Aachen, Germany
| | - Abin Jose
- Faculty of Electrical Engineering and Information Technology, RWTH Aachen University, Aachen, Germany
| | - Rijo Roy
- Faculty of Electrical Engineering and Information Technology, RWTH Aachen University, Aachen, Germany
| | - Dorit Merhof
- Faculty of Informatics and Data Science, University of Regensburg, Regensburg, Germany; Fraunhofer Institute for Digital Medicine MEVIS, Bremen, Germany.
| |
Collapse
|
20
|
Cao C, Huang W, Hu F, Gao X. Hierarchical neural architecture search with adaptive global-local feature learning for Magnetic Resonance Image reconstruction. Comput Biol Med 2024; 168:107774. [PMID: 38039897 DOI: 10.1016/j.compbiomed.2023.107774] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/18/2023] [Revised: 10/29/2023] [Accepted: 11/26/2023] [Indexed: 12/03/2023]
Abstract
Neural architecture search (NAS) has been introduced into the design of deep neural network architectures for Magnetic Resonance Imaging (MRI) reconstruction since NAS-based methods can acquire the complex network architecture automatically without professional designing experience and improve the model's generalization ability. However, current NAS-based MRI reconstruction methods suffer from a lack of efficient operators in the search space, which leads to challenges in effectively recovering high-frequency details. This limitation is primarily due to the prevalent use of convolution operators in the current search space, which struggle to capture both global and local features of MR images simultaneously, resulting in insufficient information utilization. To address this issue, a generative adversarial network (GAN) based model is proposed to reconstruct the MR image from under-sampled K-space data. Firstly, parameterized global and local feature learning modules at multiple scales are added into the search space to improve the capability of recovering high-frequency details. Secondly, to mitigate the increased search time caused by the augmented search space, a hierarchical NAS is designed to learn the global-local feature learning modules that enable the reconstruction network to learn global and local information of MR images at different scales adaptively. Thirdly, to reduce the number of network parameters and computational complexity, the standard operations in global-local feature learning modules are replaced with lightweight operations. Finally, experiments on several publicly available brain MRI image datasets evaluate the performance of the proposed method. Compared to the state-of-the-art MRI reconstruction methods, the proposed method yields better reconstruction results in terms of peak signal-to-noise ratio and structural similarity at a lower computational cost. Additionally, our reconstruction results are validated through a brain tumor classification task, affirming the practicability of the proposed method. Our code is available at https://github.com/wwHwo/HNASMRI.
Collapse
Affiliation(s)
- Chunhong Cao
- MOE Key Laboratory of Intelligent Computing and Information Processing, Xiangtan University, Xiangtan, 411100, China
| | - Wenwei Huang
- MOE Key Laboratory of Intelligent Computing and Information Processing, Xiangtan University, Xiangtan, 411100, China
| | - Fang Hu
- Key Laboratory of Medical Imaging and Artificial Intelligence of Hunan Province, Xiangnan University, Chenzhou, 423043, China.
| | - Xieping Gao
- Hunan Provincial Key Laboratory of Intelligent Computing and Language Information Processing, Hunan Normal University, Changsha, 410081, China.
| |
Collapse
|
21
|
Ozbey M, Dalmaz O, Dar SUH, Bedel HA, Ozturk S, Gungor A, Cukur T. Unsupervised Medical Image Translation With Adversarial Diffusion Models. IEEE TRANSACTIONS ON MEDICAL IMAGING 2023; 42:3524-3539. [PMID: 37379177 DOI: 10.1109/tmi.2023.3290149] [Citation(s) in RCA: 20] [Impact Index Per Article: 20.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/30/2023]
Abstract
Imputation of missing images via source-to-target modality translation can improve diversity in medical imaging protocols. A pervasive approach for synthesizing target images involves one-shot mapping through generative adversarial networks (GAN). Yet, GAN models that implicitly characterize the image distribution can suffer from limited sample fidelity. Here, we propose a novel method based on adversarial diffusion modeling, SynDiff, for improved performance in medical image translation. To capture a direct correlate of the image distribution, SynDiff leverages a conditional diffusion process that progressively maps noise and source images onto the target image. For fast and accurate image sampling during inference, large diffusion steps are taken with adversarial projections in the reverse diffusion direction. To enable training on unpaired datasets, a cycle-consistent architecture is devised with coupled diffusive and non-diffusive modules that bilaterally translate between two modalities. Extensive assessments are reported on the utility of SynDiff against competing GAN and diffusion models in multi-contrast MRI and MRI-CT translation. Our demonstrations indicate that SynDiff offers quantitatively and qualitatively superior performance against competing baselines.
Collapse
|
22
|
Yan Y, Yang T, Zhao X, Jiao C, Yang A, Miao J. DC-SiamNet: Deep contrastive Siamese network for self-supervised MRI reconstruction. Comput Biol Med 2023; 167:107619. [PMID: 37925909 DOI: 10.1016/j.compbiomed.2023.107619] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/22/2023] [Revised: 10/03/2023] [Accepted: 10/23/2023] [Indexed: 11/07/2023]
Abstract
Reconstruction methods based on deep learning have greatly shortened the data acquisition time of magnetic resonance imaging (MRI). However, these methods typically utilize massive fully sampled data for supervised training, restricting their application in certain clinical scenarios and posing challenges to the reconstruction effect when high-quality MR images are unavailable. Recently, self-supervised methods have been developed that only undersampled MRI images participate in the network training. Nevertheless, due to the lack of complete referable MR image data, self-supervised reconstruction is prone to produce incorrect structure contents, such as unnatural texture details and over-smoothed tissue sites. To solve this problem, we propose a self-supervised Deep Contrastive Siamese Network (DC-SiamNet) for fast MR imaging. First, DC-SiamNet performs the reconstruction with a Siamese unrolled structure and obtains visual representations in different iterative phases. Particularly, an attention-weighted average pooling module is employed at the bottleneck layer of the U-shape regularization unit, which can effectively aggregate valuable local information of the underlying feature map in the generated representation vector. Then, a novel hybrid loss function is designed to drive the self-supervised reconstruction and contrastive learning simultaneously by forcing the output consistency across different branches in the frequency domain, the image domain, and the latent space. The proposed method is extensively evaluated with different sampling patterns on the IXI brain dataset and the MRINet knee dataset. Experimental results show that DC-SiamNet can achieve 0.93 in structural similarity and 33.984 dB in peak signal-to-noise ratio on the IXI brain dataset under 8x acceleration. It has better reconstruction accuracy than other methods, and the performance is close to the corresponding model trained with full supervision, especially when the sampling rate is low. In addition, generalization experiments verify that our method has a strong cross-domain reconstruction ability for different contrast brain images.
Collapse
Affiliation(s)
- Yanghui Yan
- School of Information Science and Engineering, Henan University of Technology, Zhengzhou, 450001, China
| | - Tiejun Yang
- School of Artificial Intelligence and Big Data, Henan University of Technology, Zhengzhou, 450001, China; Key Laboratory of Grain Information Processing and Control (HAUT), Ministry of Education, Zhengzhou, China; Henan Key Laboratory of Grain Photoelectric Detection and Control (HAUT), Zhengzhou, Henan, China.
| | - Xiang Zhao
- School of Information Science and Engineering, Henan University of Technology, Zhengzhou, 450001, China
| | - Chunxia Jiao
- School of Information Science and Engineering, Henan University of Technology, Zhengzhou, 450001, China
| | - Aolin Yang
- School of Information Science and Engineering, Henan University of Technology, Zhengzhou, 450001, China
| | - Jianyu Miao
- School of Artificial Intelligence and Big Data, Henan University of Technology, Zhengzhou, 450001, China
| |
Collapse
|
23
|
Sharma R, Tsiamyrtzis P, Webb AG, Leiss EL, Tsekos NV. Learning to deep learning: statistics and a paradigm test in selecting a UNet architecture to enhance MRI. MAGMA (NEW YORK, N.Y.) 2023:10.1007/s10334-023-01127-6. [PMID: 37989921 DOI: 10.1007/s10334-023-01127-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/05/2023] [Revised: 09/30/2023] [Accepted: 10/16/2023] [Indexed: 11/23/2023]
Abstract
OBJECTIVE This study aims to assess the statistical significance of training parameters in 240 dense UNets (DUNets) used for enhancing low Signal-to-Noise Ratio (SNR) and undersampled MRI in various acquisition protocols. The objective is to determine the validity of differences between different DUNet configurations and their impact on image quality metrics. MATERIALS AND METHODS To achieve this, we trained all DUNets using the same learning rate and number of epochs, with variations in 5 acquisition protocols, 24 loss function weightings, and 2 ground truths. We calculated evaluation metrics for two metric regions of interest (ROI). We employed both Analysis of Variance (ANOVA) and Mixed Effects Model (MEM) to assess the statistical significance of the independent parameters, aiming to compare their efficacy in revealing differences and interactions among fixed parameters. RESULTS ANOVA analysis showed that, except for the acquisition protocol, fixed variables were statistically insignificant. In contrast, MEM analysis revealed that all fixed parameters and their interactions held statistical significance. This emphasizes the need for advanced statistical analysis in comparative studies, where MEM can uncover finer distinctions often overlooked by ANOVA. DISCUSSION These findings highlight the importance of utilizing appropriate statistical analysis when comparing different deep learning models. Additionally, the surprising effectiveness of the UNet architecture in enhancing various acquisition protocols underscores the potential for developing improved methods for characterizing and training deep learning models. This study serves as a stepping stone toward enhancing the transparency and comparability of deep learning techniques for medical imaging applications.
Collapse
Affiliation(s)
- Rishabh Sharma
- Medical Robotics and Imaging Lab, Department of Computer Science, 501, Philip G. Hoffman Hall, University of Houston, 4800 Calhoun Road, Houston, TX, 77204, USA
| | - Panagiotis Tsiamyrtzis
- Department of Mechanical Engineering, Politecnico Di Milano, Milan, Italy
- Department of Statistics, Athens University of Economics and Business, Athens, Greece
| | - Andrew G Webb
- C.J. Gorter Center for High Field MRI, Department of Radiology, Leiden University Medical Center, Leiden, The Netherlands
| | - Ernst L Leiss
- Department of Computer Science, University of Houston, Houston, TX, USA
| | - Nikolaos V Tsekos
- Medical Robotics and Imaging Lab, Department of Computer Science, 501, Philip G. Hoffman Hall, University of Houston, 4800 Calhoun Road, Houston, TX, 77204, USA.
| |
Collapse
|
24
|
Shafique M, Qazi SA, Omer H. Compressed SVD-based L + S model to reconstruct undersampled dynamic MRI data using parallel architecture. MAGMA (NEW YORK, N.Y.) 2023:10.1007/s10334-023-01128-5. [PMID: 37978992 DOI: 10.1007/s10334-023-01128-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/27/2023] [Revised: 09/27/2023] [Accepted: 10/20/2023] [Indexed: 11/19/2023]
Abstract
BACKGROUND Magnetic Resonance Imaging (MRI) is a highly demanded medical imaging system due to high resolution, large volumetric coverage, and ability to capture the dynamic and functional information of body organs e.g. cardiac MRI is employed to assess cardiac structure and evaluate blood flow dynamics through the cardiac valves. Long scan time is the main drawback of MRI, which makes it difficult for the patients to remain still during the scanning process. OBJECTIVE By collecting fewer measurements, MRI scan time can be shortened, but this undersampling causes aliasing artifacts in the reconstructed images. Advanced image reconstruction algorithms have been used in literature to overcome these undersampling artifacts. These algorithms are computationally expensive and require a long time for reconstruction which makes them infeasible for real-time clinical applications e.g. cardiac MRI. However, exploiting the inherent parallelism in these algorithms can help to reduce their computation time. METHODS Low-rank plus sparse (L+S) matrix decomposition model is a technique used in literature to reconstruct the highly undersampled dynamic MRI (dMRI) data at the expense of long reconstruction time. In this paper, Compressed Singular Value Decomposition (cSVD) model is used in L+S decomposition model (instead of conventional SVD) to reduce the reconstruction time. The results provide improved quality of the reconstructed images. Furthermore, it has been observed that cSVD and other parts of the L+S model possess highly parallel operations; therefore, a customized GPU based parallel architecture of the modified L+S model has been presented to further reduce the reconstruction time. RESULTS Four cardiac MRI datasets (three different cardiac perfusion acquired from different patients and one cardiac cine data), each with different acceleration factors of 2, 6 and 8 are used for experiments in this paper. Experimental results demonstrate that using the proposed parallel architecture for the reconstruction of cardiac perfusion data provides a speed-up factor up to 19.15× (with memory latency) and 70.55× (without memory latency) in comparison to the conventional CPU reconstruction with no compromise on image quality. CONCLUSION The proposed method is well-suited for real-time clinical applications, offering a substantial reduction in reconstruction time.
Collapse
Affiliation(s)
- Muhammad Shafique
- Medical Image Processing Research Group (MIPRG), Department of Electrical and Computer Engineering, COMSATS University Islamabad, Islamabad, Pakistan.
- Department of Electrical Engineering, University of Poonch Rawalakot, Rawalakot, AJ&K, Pakistan.
| | - Sohaib Ayaz Qazi
- Cardiovascular Sciences, Department of Health, Medicine and Caring Sciences, Linköping University, Linköping, Sweden
- Center for Medical Image Science and Visualization (CMIV), Linköping University, Linköping, Sweden
| | - Hammad Omer
- Medical Image Processing Research Group (MIPRG), Department of Electrical and Computer Engineering, COMSATS University Islamabad, Islamabad, Pakistan
| |
Collapse
|
25
|
Ahmed H, Zhang Q, Wong F, Donnan R, Alomainy A. Lesion Detection in Optical Coherence Tomography with Transformer-Enhanced Detector. J Imaging 2023; 9:244. [PMID: 37998091 PMCID: PMC10671998 DOI: 10.3390/jimaging9110244] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/04/2023] [Revised: 10/10/2023] [Accepted: 10/11/2023] [Indexed: 11/25/2023] Open
Abstract
Optical coherence tomography (OCT) is an emerging imaging tool in healthcare with common applications in ophthalmology for the detection of retinal diseases and in dentistry for the early detection of tooth decay. Speckle noise is ubiquitous in OCT images, which can hinder diagnosis by clinicians. In this paper, a region-based, deep learning framework for the detection of anomalies is proposed for OCT-acquired images. The core of the framework is Transformer-Enhanced Detection (TED), which includes attention gates (AGs) to ensure focus is placed on the foreground while identifying and removing noise artifacts as anomalies. TED was designed to detect the different types of anomalies commonly present in OCT images for diagnostic purposes and thus aid clinical interpretation. Extensive quantitative evaluations were performed to measure the performance of TED against current, widely known, deep learning detection algorithms. Three different datasets were tested: two dental and one CT (hosting scans of lung nodules, livers, etc.). The results showed that the approach verifiably detected tooth decay and numerous lesions across two modalities, achieving superior performance compared to several well-known algorithms. The proposed method improved the accuracy of detection by 16-22% and the Intersection over Union (IOU) by 10% for both dentistry datasets. For the CT dataset, the performance metrics were similarly improved by 9% and 20%, respectively.
Collapse
Affiliation(s)
- Hanya Ahmed
- Department of Electronic Engineering and Computer Science, Queen Mary University of London—QMUL, London E1 4NS, UK (R.D.); (A.A.)
| | - Qianni Zhang
- Department of Electronic Engineering and Computer Science, Queen Mary University of London—QMUL, London E1 4NS, UK (R.D.); (A.A.)
| | - Ferranti Wong
- Institute of Dentistry at Barts Health, Queen Mary University of London—QMUL, London E1 4NS, UK
| | - Robert Donnan
- Department of Electronic Engineering and Computer Science, Queen Mary University of London—QMUL, London E1 4NS, UK (R.D.); (A.A.)
| | - Akram Alomainy
- Department of Electronic Engineering and Computer Science, Queen Mary University of London—QMUL, London E1 4NS, UK (R.D.); (A.A.)
| |
Collapse
|
26
|
Peng H, Jiang C, Cheng J, Zhang M, Wang S, Liang D, Liu Q. One-Shot Generative Prior in Hankel-k-Space for Parallel Imaging Reconstruction. IEEE TRANSACTIONS ON MEDICAL IMAGING 2023; 42:3420-3435. [PMID: 37342955 DOI: 10.1109/tmi.2023.3288219] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/23/2023]
Abstract
Magnetic resonance imaging serves as an essential tool for clinical diagnosis. However, it suffers from a long acquisition time. The utilization of deep learning, especially the deep generative models, offers aggressive acceleration and better reconstruction in magnetic resonance imaging. Nevertheless, learning the data distribution as prior knowledge and reconstructing the image from limited data remains challenging. In this work, we propose a novel Hankel-k-space generative model (HKGM), which can generate samples from a training set of as little as one k-space. At the prior learning stage, we first construct a large Hankel matrix from k-space data, then extract multiple structured k-space patches from the Hankel matrix to capture the internal distribution among different patches. Extracting patches from a Hankel matrix enables the generative model to be learned from the redundant and low-rank data space. At the iterative reconstruction stage, the desired solution obeys the learned prior knowledge. The intermediate reconstruction solution is updated by taking it as the input of the generative model. The updated result is then alternatively operated by imposing low-rank penalty on its Hankel matrix and data consistency constraint on the measurement data. Experimental results confirmed that the internal statistics of patches within single k-space data carry enough information for learning a powerful generative model and providing state-of-the-art reconstruction.
Collapse
|
27
|
Pramanik A, Bhave S, Sajib S, Sharma SD, Jacob M. Adapting model-based deep learning to multiple acquisition conditions: Ada-MoDL. Magn Reson Med 2023; 90:2033-2051. [PMID: 37332189 PMCID: PMC10524947 DOI: 10.1002/mrm.29750] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/31/2022] [Revised: 05/21/2023] [Accepted: 05/22/2023] [Indexed: 06/20/2023]
Abstract
PURPOSE The aim of this work is to introduce a single model-based deep network that can provide high-quality reconstructions from undersampled parallel MRI data acquired with multiple sequences, acquisition settings, and field strengths. METHODS A single unrolled architecture, which offers good reconstructions for multiple acquisition settings, is introduced. The proposed scheme adapts the model to each setting by scaling the convolutional neural network (CNN) features and the regularization parameter with appropriate weights. The scaling weights and regularization parameter are derived using a multilayer perceptron model from conditional vectors, which represents the specific acquisition setting. The perceptron parameters and the CNN weights are jointly trained using data from multiple acquisition settings, including differences in field strengths, acceleration, and contrasts. The conditional network is validated using datasets acquired with different acquisition settings. RESULTS The comparison of the adaptive framework, which trains a single model using the data from all the settings, shows that it can offer consistently improved performance for each acquisition condition. The comparison of the proposed scheme with networks that are trained independently for each acquisition setting shows that it requires less training data per acquisition setting to offer good performance. CONCLUSION The Ada-MoDL framework enables the use of a single model-based unrolled network for multiple acquisition settings. In addition to eliminating the need to train and store multiple networks for different acquisition settings, this approach reduces the training data needed for each acquisition setting.
Collapse
Affiliation(s)
- Aniket Pramanik
- Department of Electrical and Computer Engineering, University of Iowa, Iowa, USA
| | - Sampada Bhave
- Canon Medical Research USA, Inc., Mayfield Village, Ohio, USA
| | - Saurav Sajib
- Canon Medical Research USA, Inc., Mayfield Village, Ohio, USA
| | - Samir D. Sharma
- Canon Medical Research USA, Inc., Mayfield Village, Ohio, USA
| | - Mathews Jacob
- Department of Electrical and Computer Engineering, University of Iowa, Iowa, USA
| |
Collapse
|
28
|
Kim G, Baek J. Power-law spectrum-based objective function to train a generative adversarial network with transfer learning for the synthetic breast CT image. Phys Med Biol 2023; 68:205007. [PMID: 37722388 DOI: 10.1088/1361-6560/acfadf] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/02/2023] [Accepted: 09/18/2023] [Indexed: 09/20/2023]
Abstract
Objective.This paper proposes a new objective function to improve the quality of synthesized breast CT images generated by the GAN and compares the GAN performances on transfer learning datasets from different image domains.Approach.The proposed objective function, named beta loss function, is based on the fact that x-ray-based breast images follow the power-law spectrum. Accordingly, the exponent of the power-law spectrum (beta value) for breast CT images is approximately two. The beta loss function is defined in terms of L1 distance between the beta value of synthetic images and validation samples. To compare the GAN performances for transfer learning datasets from different image domains, ImageNet and anatomical noise images are used in the transfer learning dataset. We employ styleGAN2 as the backbone network and add the proposed beta loss function. The patient-derived breast CT dataset is used as the training and validation dataset; 7355 and 212 images are used for network training and validation, respectively. We use the beta value evaluation and Fréchet inception distance (FID) score for quantitative evaluation.Main results.For qualitative assessment, we attempt to replicate the images from the validation dataset using the trained GAN. Our results show that the proposed beta loss function achieves a more similar beta value to real images and a lower FID score. Moreover, we observe that the GAN pretrained with anatomical noise images achieves better equality than ImageNet for beta value evaluation and FID score. Finally, the beta loss function with anatomical noise as the transfer learning dataset achieves the lowest FID score.Significance.Overall, the GAN using the proposed beta loss function with anatomical noise images as the transfer learning dataset provides the lowest FID score among all tested cases. Hence, this work has implications for developing GAN-based breast image synthesis methods for medical imaging applications.
Collapse
Affiliation(s)
- Gihun Kim
- School of Integrated Technology, Yonsei University, Republic of Korea
| | - Jongduk Baek
- Department of Artificial Intelligence, Yonsei University, Republic of Korea
- Baruenex Imaging, Republic of Korea
| |
Collapse
|
29
|
Liu J, Pasumarthi S, Duffy B, Gong E, Datta K, Zaharchuk G. One Model to Synthesize Them All: Multi-Contrast Multi-Scale Transformer for Missing Data Imputation. IEEE TRANSACTIONS ON MEDICAL IMAGING 2023; 42:2577-2591. [PMID: 37030684 PMCID: PMC10543020 DOI: 10.1109/tmi.2023.3261707] [Citation(s) in RCA: 8] [Impact Index Per Article: 8.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/19/2023]
Abstract
Multi-contrast magnetic resonance imaging (MRI) is widely used in clinical practice as each contrast provides complementary information. However, the availability of each imaging contrast may vary amongst patients, which poses challenges to radiologists and automated image analysis algorithms. A general approach for tackling this problem is missing data imputation, which aims to synthesize the missing contrasts from existing ones. While several convolutional neural networks (CNN) based algorithms have been proposed, they suffer from the fundamental limitations of CNN models, such as the requirement for fixed numbers of input and output channels, the inability to capture long-range dependencies, and the lack of interpretability. In this work, we formulate missing data imputation as a sequence-to-sequence learning problem and propose a multi-contrast multi-scale Transformer (MMT), which can take any subset of input contrasts and synthesize those that are missing. MMT consists of a multi-scale Transformer encoder that builds hierarchical representations of inputs combined with a multi-scale Transformer decoder that generates the outputs in a coarse-to-fine fashion. The proposed multi-contrast Swin Transformer blocks can efficiently capture intra- and inter-contrast dependencies for accurate image synthesis. Moreover, MMT is inherently interpretable as it allows us to understand the importance of each input contrast in different regions by analyzing the in-built attention maps of Transformer blocks in the decoder. Extensive experiments on two large-scale multi-contrast MRI datasets demonstrate that MMT outperforms the state-of-the-art methods quantitatively and qualitatively.
Collapse
|
30
|
Wang N, Lin S, Li X, Li K, Shen Y, Gao Y, Ma L. MISSU: 3D Medical Image Segmentation via Self-Distilling TransUNet. IEEE TRANSACTIONS ON MEDICAL IMAGING 2023; 42:2740-2750. [PMID: 37018113 DOI: 10.1109/tmi.2023.3264433] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/19/2023]
Abstract
U-Nets have achieved tremendous success in medical image segmentation. Nevertheless, it may have limitations in global (long-range) contextual interactions and edge-detail preservation. In contrast, the Transformer module has an excellent ability to capture long-range dependencies by leveraging the self-attention mechanism into the encoder. Although the Transformer module was born to model the long-range dependency on the extracted feature maps, it still suffers high computational and spatial complexities in processing high-resolution 3D feature maps. This motivates us to design an efficient Transformer-based UNet model and study the feasibility of Transformer-based network architectures for medical image segmentation tasks. To this end, we propose to self-distill a Transformer-based UNet for medical image segmentation, which simultaneously learns global semantic information and local spatial-detailed features. Meanwhile, a local multi-scale fusion block is first proposed to refine fine-grained details from the skipped connections in the encoder by the main CNN stem through self-distillation, only computed during training and removed at inference with minimal overhead. Extensive experiments on BraTS 2019 and CHAOS datasets show that our MISSU achieves the best performance over previous state-of-the-art methods. Code and models are available at: https://github.com/wangn123/MISSU.git.
Collapse
|
31
|
Singh D, Monga A, de Moura HL, Zhang X, Zibetti MVW, Regatte RR. Emerging Trends in Fast MRI Using Deep-Learning Reconstruction on Undersampled k-Space Data: A Systematic Review. Bioengineering (Basel) 2023; 10:1012. [PMID: 37760114 PMCID: PMC10525988 DOI: 10.3390/bioengineering10091012] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/31/2023] [Revised: 08/22/2023] [Accepted: 08/24/2023] [Indexed: 09/29/2023] Open
Abstract
Magnetic Resonance Imaging (MRI) is an essential medical imaging modality that provides excellent soft-tissue contrast and high-resolution images of the human body, allowing us to understand detailed information on morphology, structural integrity, and physiologic processes. However, MRI exams usually require lengthy acquisition times. Methods such as parallel MRI and Compressive Sensing (CS) have significantly reduced the MRI acquisition time by acquiring less data through undersampling k-space. The state-of-the-art of fast MRI has recently been redefined by integrating Deep Learning (DL) models with these undersampled approaches. This Systematic Literature Review (SLR) comprehensively analyzes deep MRI reconstruction models, emphasizing the key elements of recently proposed methods and highlighting their strengths and weaknesses. This SLR involves searching and selecting relevant studies from various databases, including Web of Science and Scopus, followed by a rigorous screening and data extraction process using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. It focuses on various techniques, such as residual learning, image representation using encoders and decoders, data-consistency layers, unrolled networks, learned activations, attention modules, plug-and-play priors, diffusion models, and Bayesian methods. This SLR also discusses the use of loss functions and training with adversarial networks to enhance deep MRI reconstruction methods. Moreover, we explore various MRI reconstruction applications, including non-Cartesian reconstruction, super-resolution, dynamic MRI, joint learning of reconstruction with coil sensitivity and sampling, quantitative mapping, and MR fingerprinting. This paper also addresses research questions, provides insights for future directions, and emphasizes robust generalization and artifact handling. Therefore, this SLR serves as a valuable resource for advancing fast MRI, guiding research and development efforts of MRI reconstruction for better image quality and faster data acquisition.
Collapse
Affiliation(s)
- Dilbag Singh
- Center of Biomedical Imaging, Department of Radiology, New York University Grossman School of Medicine, New York, NY 10016, USA; (A.M.); (H.L.d.M.); (X.Z.); (M.V.W.Z.)
| | | | | | | | | | - Ravinder R. Regatte
- Center of Biomedical Imaging, Department of Radiology, New York University Grossman School of Medicine, New York, NY 10016, USA; (A.M.); (H.L.d.M.); (X.Z.); (M.V.W.Z.)
| |
Collapse
|
32
|
Kazerouni A, Aghdam EK, Heidari M, Azad R, Fayyaz M, Hacihaliloglu I, Merhof D. Diffusion models in medical imaging: A comprehensive survey. Med Image Anal 2023; 88:102846. [PMID: 37295311 DOI: 10.1016/j.media.2023.102846] [Citation(s) in RCA: 29] [Impact Index Per Article: 29.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/16/2022] [Revised: 05/12/2023] [Accepted: 05/16/2023] [Indexed: 06/12/2023]
Abstract
Denoising diffusion models, a class of generative models, have garnered immense interest lately in various deep-learning problems. A diffusion probabilistic model defines a forward diffusion stage where the input data is gradually perturbed over several steps by adding Gaussian noise and then learns to reverse the diffusion process to retrieve the desired noise-free data from noisy data samples. Diffusion models are widely appreciated for their strong mode coverage and quality of the generated samples in spite of their known computational burdens. Capitalizing on the advances in computer vision, the field of medical imaging has also observed a growing interest in diffusion models. With the aim of helping the researcher navigate this profusion, this survey intends to provide a comprehensive overview of diffusion models in the discipline of medical imaging. Specifically, we start with an introduction to the solid theoretical foundation and fundamental concepts behind diffusion models and the three generic diffusion modeling frameworks, namely, diffusion probabilistic models, noise-conditioned score networks, and stochastic differential equations. Then, we provide a systematic taxonomy of diffusion models in the medical domain and propose a multi-perspective categorization based on their application, imaging modality, organ of interest, and algorithms. To this end, we cover extensive applications of diffusion models in the medical domain, including image-to-image translation, reconstruction, registration, classification, segmentation, denoising, 2/3D generation, anomaly detection, and other medically-related challenges. Furthermore, we emphasize the practical use case of some selected approaches, and then we discuss the limitations of the diffusion models in the medical domain and propose several directions to fulfill the demands of this field. Finally, we gather the overviewed studies with their available open-source implementations at our GitHub.1 We aim to update the relevant latest papers within it regularly.
Collapse
Affiliation(s)
- Amirhossein Kazerouni
- School of Electrical Engineering, Iran University of Science and Technology, Tehran, Iran
| | | | - Moein Heidari
- School of Electrical Engineering, Iran University of Science and Technology, Tehran, Iran
| | - Reza Azad
- Faculty of Electrical Engineering and Information Technology, RWTH Aachen University, Aachen, Germany
| | | | - Ilker Hacihaliloglu
- Department of Radiology, University of British Columbia, Vancouver, Canada; Department of Medicine, University of British Columbia, Vancouver, Canada
| | - Dorit Merhof
- Faculty of Informatics and Data Science, University of Regensburg, Regensburg, Germany; Fraunhofer Institute for Digital Medicine MEVIS, Bremen, Germany.
| |
Collapse
|
33
|
Al-Hammuri K, Gebali F, Kanan A, Chelvan IT. Vision transformer architecture and applications in digital health: a tutorial and survey. Vis Comput Ind Biomed Art 2023; 6:14. [PMID: 37428360 DOI: 10.1186/s42492-023-00140-9] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/21/2023] [Accepted: 05/30/2023] [Indexed: 07/11/2023] Open
Abstract
The vision transformer (ViT) is a state-of-the-art architecture for image recognition tasks that plays an important role in digital health applications. Medical images account for 90% of the data in digital medicine applications. This article discusses the core foundations of the ViT architecture and its digital health applications. These applications include image segmentation, classification, detection, prediction, reconstruction, synthesis, and telehealth such as report generation and security. This article also presents a roadmap for implementing the ViT in digital health systems and discusses its limitations and challenges.
Collapse
Affiliation(s)
- Khalid Al-Hammuri
- Electrical and Computer Engineering, University of Victoria, Victoria, V8W 2Y2, Canada.
| | - Fayez Gebali
- Electrical and Computer Engineering, University of Victoria, Victoria, V8W 2Y2, Canada
| | - Awos Kanan
- Computer Engineering, Princess Sumaya University for Technology, Amman, 11941, Jordan
| | | |
Collapse
|
34
|
La Greca Saint-Esteven A, Dal Bello R, Lapaeva M, Fankhauser L, Pouymayou B, Konukoglu E, Andratschke N, Balermpas P, Guckenberger M, Tanadini-Lang S. Synthetic computed tomography for low-field magnetic resonance-only radiotherapy in head-and-neck cancer using residual vision transformers. Phys Imaging Radiat Oncol 2023; 27:100471. [PMID: 37497191 PMCID: PMC10366636 DOI: 10.1016/j.phro.2023.100471] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/01/2023] [Revised: 07/06/2023] [Accepted: 07/06/2023] [Indexed: 07/28/2023] Open
Abstract
Background and purpose Synthetic computed tomography (sCT) scans are necessary for dose calculation in magnetic resonance (MR)-only radiotherapy. While deep learning (DL) has shown remarkable performance in generating sCT scans from MR images, research has predominantly focused on high-field MR images. This study presents the first implementation of a DL model for sCT generation in head-and-neck (HN) cancer using low-field MR images. Specifically, the use of vision transformers (ViTs) was explored. Materials and methods The dataset consisted of 31 patients, resulting in 196 pairs of deformably-registered computed tomography (dCT) and MR scans. The latter were obtained using a balanced steady-state precession sequence on a 0.35T scanner. Residual ViTs were trained on 2D axial, sagittal, and coronal slices, respectively, and the final sCTs were generated by averaging the models' outputs. Different image similarity metrics, dose volume histogram (DVH) deviations, and gamma analyses were computed on the test set (n = 6). The overlap between auto-contours on sCT scans and manual contours on MR images was evaluated for different organs-at-risk using the Dice score. Results The median [range] value of the test mean absolute error was 57 [37-74] HU. DVH deviations were below 1% for all structures. The median gamma passing rates exceeded 94% in the 2%/2mm analysis (threshold = 90%). The median Dice scores were above 0.7 for all organs-at-risk. Conclusions The clinical applicability of DL-based sCT generation from low-field MR images in HN cancer was proved. High sCT-dCT similarity and dose metric accuracy were achieved, and sCT suitability for organs-at-risk auto-delineation was shown.
Collapse
Affiliation(s)
- Agustina La Greca Saint-Esteven
- Department of Radiation Oncology, University Hospital Zurich and University of Zurich, Rämistrasse 100, Zurich 8091, Switzerland
- Computer Vision Laboratory, Department of Information Technology and Electrical Engineering, ETH Zurich, Sternwartstrasse 7, Zurich 8092, Switzerland
| | - Ricardo Dal Bello
- Department of Radiation Oncology, University Hospital Zurich and University of Zurich, Rämistrasse 100, Zurich 8091, Switzerland
| | - Mariia Lapaeva
- Department of Radiation Oncology, University Hospital Zurich and University of Zurich, Rämistrasse 100, Zurich 8091, Switzerland
| | - Lisa Fankhauser
- Department of Radiation Oncology, University Hospital Zurich and University of Zurich, Rämistrasse 100, Zurich 8091, Switzerland
| | - Bertrand Pouymayou
- Department of Radiation Oncology, University Hospital Zurich and University of Zurich, Rämistrasse 100, Zurich 8091, Switzerland
| | - Ender Konukoglu
- Computer Vision Laboratory, Department of Information Technology and Electrical Engineering, ETH Zurich, Sternwartstrasse 7, Zurich 8092, Switzerland
| | - Nicolaus Andratschke
- Department of Radiation Oncology, University Hospital Zurich and University of Zurich, Rämistrasse 100, Zurich 8091, Switzerland
| | - Panagiotis Balermpas
- Department of Radiation Oncology, University Hospital Zurich and University of Zurich, Rämistrasse 100, Zurich 8091, Switzerland
| | - Matthias Guckenberger
- Department of Radiation Oncology, University Hospital Zurich and University of Zurich, Rämistrasse 100, Zurich 8091, Switzerland
| | - Stephanie Tanadini-Lang
- Department of Radiation Oncology, University Hospital Zurich and University of Zurich, Rämistrasse 100, Zurich 8091, Switzerland
| |
Collapse
|
35
|
Güngör A, Dar SU, Öztürk Ş, Korkmaz Y, Bedel HA, Elmas G, Ozbey M, Çukur T. Adaptive diffusion priors for accelerated MRI reconstruction. Med Image Anal 2023; 88:102872. [PMID: 37384951 DOI: 10.1016/j.media.2023.102872] [Citation(s) in RCA: 21] [Impact Index Per Article: 21.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/31/2022] [Revised: 04/13/2023] [Accepted: 06/12/2023] [Indexed: 07/01/2023]
Abstract
Deep MRI reconstruction is commonly performed with conditional models that de-alias undersampled acquisitions to recover images consistent with fully-sampled data. Since conditional models are trained with knowledge of the imaging operator, they can show poor generalization across variable operators. Unconditional models instead learn generative image priors decoupled from the operator to improve reliability against domain shifts related to the imaging operator. Recent diffusion models are particularly promising given their high sample fidelity. Nevertheless, inference with a static image prior can perform suboptimally. Here we propose the first adaptive diffusion prior for MRI reconstruction, AdaDiff, to improve performance and reliability against domain shifts. AdaDiff leverages an efficient diffusion prior trained via adversarial mapping over large reverse diffusion steps. A two-phase reconstruction is executed following training: a rapid-diffusion phase that produces an initial reconstruction with the trained prior, and an adaptation phase that further refines the result by updating the prior to minimize data-consistency loss. Demonstrations on multi-contrast brain MRI clearly indicate that AdaDiff outperforms competing conditional and unconditional methods under domain shifts, and achieves superior or on par within-domain performance.
Collapse
Affiliation(s)
- Alper Güngör
- Department of Electrical and Electronics Engineering, Bilkent University, Ankara 06800, Turkey; National Magnetic Resonance Research Center (UMRAM), Bilkent University, Ankara 06800, Turkey; ASELSAN Research Center, Ankara 06200, Turkey
| | - Salman Uh Dar
- Department of Electrical and Electronics Engineering, Bilkent University, Ankara 06800, Turkey; National Magnetic Resonance Research Center (UMRAM), Bilkent University, Ankara 06800, Turkey; Department of Internal Medicine III, Heidelberg University Hospital, Heidelberg 69120, Germany
| | - Şaban Öztürk
- Department of Electrical and Electronics Engineering, Bilkent University, Ankara 06800, Turkey; National Magnetic Resonance Research Center (UMRAM), Bilkent University, Ankara 06800, Turkey; Department of Electrical and Electronics Engineering, Amasya University, Amasya 05100, Turkey
| | - Yilmaz Korkmaz
- Department of Electrical and Electronics Engineering, Bilkent University, Ankara 06800, Turkey; National Magnetic Resonance Research Center (UMRAM), Bilkent University, Ankara 06800, Turkey
| | - Hasan A Bedel
- Department of Electrical and Electronics Engineering, Bilkent University, Ankara 06800, Turkey; National Magnetic Resonance Research Center (UMRAM), Bilkent University, Ankara 06800, Turkey
| | - Gokberk Elmas
- Department of Electrical and Electronics Engineering, Bilkent University, Ankara 06800, Turkey; National Magnetic Resonance Research Center (UMRAM), Bilkent University, Ankara 06800, Turkey
| | - Muzaffer Ozbey
- Department of Electrical and Electronics Engineering, Bilkent University, Ankara 06800, Turkey; National Magnetic Resonance Research Center (UMRAM), Bilkent University, Ankara 06800, Turkey
| | - Tolga Çukur
- Department of Electrical and Electronics Engineering, Bilkent University, Ankara 06800, Turkey; National Magnetic Resonance Research Center (UMRAM), Bilkent University, Ankara 06800, Turkey; Neuroscience Program, Bilkent University, Ankara 06800, Turkey.
| |
Collapse
|
36
|
de Souza DAR, Mathieu H, Deloulme JC, Barbier EL. Evaluation of kernel low-rank compressed sensing in preclinical diffusion magnetic resonance imaging. Front Neurosci 2023; 17:1172830. [PMID: 37332879 PMCID: PMC10272537 DOI: 10.3389/fnins.2023.1172830] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/23/2023] [Accepted: 04/28/2023] [Indexed: 06/20/2023] Open
Abstract
Compressed sensing (CS) is widely used to accelerate clinical diffusion MRI acquisitions, but it is not widely used in preclinical settings yet. In this study, we optimized and compared several CS reconstruction methods for diffusion imaging. Different undersampling patterns and two reconstruction approaches were evaluated: conventional CS, based on Berkeley Advanced Reconstruction Toolbox (BART-CS) toolbox, and a new kernel low-rank (KLR)-CS, based on kernel principal component analysis and low-resolution-phase (LRP) maps. 3D CS acquisitions were performed at 9.4T using a 4-element cryocoil on mice (wild type and a MAP6 knockout). Comparison metrics were error and structural similarity index measure (SSIM) on fractional anisotropy (FA) and mean diffusivity (MD), as well as reconstructions of the anterior commissure and fornix. Acceleration factors (AF) up to 6 were considered. In the case of retrospective undersampling, the proposed KLR-CS outperformed BART-CS up to AF = 6 for FA and MD maps and tractography. For instance, for AF = 4, the maximum errors were, respectively, 8.0% for BART-CS and 4.9% for KLR-CS, considering both FA and MD in the corpus callosum. Regarding undersampled acquisitions, these maximum errors became, respectively, 10.5% for BART-CS and 7.0% for KLR-CS. This difference between simulations and acquisitions arose mainly from repetition noise, but also from differences in resonance frequency drift, signal-to-noise ratio, and in reconstruction noise. Despite this increased error, fully sampled and AF = 2 yielded comparable results for FA, MD and tractography, and AF = 4 showed minor faults. Altogether, KLR-CS based on LRP maps seems a robust approach to accelerate preclinical diffusion MRI and thereby limit the effect of the frequency drift.
Collapse
Affiliation(s)
| | - Hervé Mathieu
- Université Grenoble Alpes, INSERM, U1216, Grenoble Institut Neurosciences, Grenoble, France
- Université Grenoble Alpes, INSERM, US17, CNRS, UAR 3552, CHU Grenoble Alpes, Grenoble, France
| | | | - Emmanuel L. Barbier
- Université Grenoble Alpes, INSERM, U1216, Grenoble Institut Neurosciences, Grenoble, France
- Université Grenoble Alpes, INSERM, US17, CNRS, UAR 3552, CHU Grenoble Alpes, Grenoble, France
| |
Collapse
|
37
|
Gao Z, Guo Y, Zhang J, Zeng T, Yang G. Hierarchical Perception Adversarial Learning Framework for Compressed Sensing MRI. IEEE TRANSACTIONS ON MEDICAL IMAGING 2023; 42:1859-1874. [PMID: 37022266 DOI: 10.1109/tmi.2023.3240862] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/02/2023]
Abstract
The long acquisition time has limited the accessibility of magnetic resonance imaging (MRI) because it leads to patient discomfort and motion artifacts. Although several MRI techniques have been proposed to reduce the acquisition time, compressed sensing in magnetic resonance imaging (CS-MRI) enables fast acquisition without compromising SNR and resolution. However, existing CS-MRI methods suffer from the challenge of aliasing artifacts. This challenge results in the noise-like textures and missing the fine details, thus leading to unsatisfactory reconstruction performance. To tackle this challenge, we propose a hierarchical perception adversarial learning framework (HP-ALF). HP-ALF can perceive the image information in the hierarchical mechanism: image-level perception and patch-level perception. The former can reduce the visual perception difference in the entire image, and thus achieve aliasing artifact removal. The latter can reduce this difference in the regions of the image, and thus recover fine details. Specifically, HP-ALF achieves the hierarchical mechanism by utilizing multilevel perspective discrimination. This discrimination can provide the information from two perspectives (overall and regional) for adversarial learning. It also utilizes a global and local coherent discriminator to provide structure information to the generator during training. In addition, HP-ALF contains a context-aware learning block to effectively exploit the slice information between individual images for better reconstruction performance. The experiments validated on three datasets demonstrate the effectiveness of HP-ALF and its superiority to the comparative methods.
Collapse
|
38
|
Xu M, Ouyang Y, Yuan Z. Deep Learning Aided Neuroimaging and Brain Regulation. SENSORS (BASEL, SWITZERLAND) 2023; 23:s23114993. [PMID: 37299724 DOI: 10.3390/s23114993] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/24/2023] [Revised: 05/15/2023] [Accepted: 05/22/2023] [Indexed: 06/12/2023]
Abstract
Currently, deep learning aided medical imaging is becoming the hot spot of AI frontier application and the future development trend of precision neuroscience. This review aimed to render comprehensive and informative insights into the recent progress of deep learning and its applications in medical imaging for brain monitoring and regulation. The article starts by providing an overview of the current methods for brain imaging, highlighting their limitations and introducing the potential benefits of using deep learning techniques to overcome these limitations. Then, we further delve into the details of deep learning, explaining the basic concepts and providing examples of how it can be used in medical imaging. One of the key strengths is its thorough discussion of the different types of deep learning models that can be used in medical imaging including convolutional neural networks (CNNs), recurrent neural networks (RNNs), and generative adversarial network (GAN) assisted magnetic resonance imaging (MRI), positron emission tomography (PET)/computed tomography (CT), electroencephalography (EEG)/magnetoencephalography (MEG), optical imaging, and other imaging modalities. Overall, our review on deep learning aided medical imaging for brain monitoring and regulation provides a referrable glance for the intersection of deep learning aided neuroimaging and brain regulation.
Collapse
Affiliation(s)
- Mengze Xu
- Center for Cognition and Neuroergonomics, State Key Laboratory of Cognitive Neuroscience and Learning, Beijing Normal University, Zhuhai 519087, China
- Centre for Cognitive and Brain Sciences, Institute of Collaborative Innovation, University of Macau, Macau SAR 999078, China
| | - Yuanyuan Ouyang
- Nanomicro Sino-Europe Technology Company Limited, Zhuhai 519031, China
- Jiangfeng China-Portugal Technology Co., Ltd., Macau SAR 999078, China
| | - Zhen Yuan
- Centre for Cognitive and Brain Sciences, Institute of Collaborative Innovation, University of Macau, Macau SAR 999078, China
| |
Collapse
|
39
|
Bedel HA, Sivgin I, Dalmaz O, Dar SUH, Çukur T. BolT: Fused window transformers for fMRI time series analysis. Med Image Anal 2023; 88:102841. [PMID: 37224718 DOI: 10.1016/j.media.2023.102841] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/19/2022] [Revised: 02/07/2023] [Accepted: 05/10/2023] [Indexed: 05/26/2023]
Abstract
Deep-learning models have enabled performance leaps in analysis of high-dimensional functional MRI (fMRI) data. Yet, many previous methods are suboptimally sensitive for contextual representations across diverse time scales. Here, we present BolT, a blood-oxygen-level-dependent transformer model, for analyzing multi-variate fMRI time series. BolT leverages a cascade of transformer encoders equipped with a novel fused window attention mechanism. Encoding is performed on temporally-overlapped windows within the time series to capture local representations. To integrate information temporally, cross-window attention is computed between base tokens in each window and fringe tokens from neighboring windows. To gradually transition from local to global representations, the extent of window overlap and thereby number of fringe tokens are progressively increased across the cascade. Finally, a novel cross-window regularization is employed to align high-level classification features across the time series. Comprehensive experiments on large-scale public datasets demonstrate the superior performance of BolT against state-of-the-art methods. Furthermore, explanatory analyses to identify landmark time points and regions that contribute most significantly to model decisions corroborate prominent neuroscientific findings in the literature.
Collapse
Affiliation(s)
- Hasan A Bedel
- Department of Electrical and Electronics Engineering, Bilkent University, Ankara 06800, Turkey; National Magnetic Resonance Research Center (UMRAM), Bilkent University, Ankara 06800, Turkey
| | - Irmak Sivgin
- Department of Electrical and Electronics Engineering, Bilkent University, Ankara 06800, Turkey; National Magnetic Resonance Research Center (UMRAM), Bilkent University, Ankara 06800, Turkey
| | - Onat Dalmaz
- Department of Electrical and Electronics Engineering, Bilkent University, Ankara 06800, Turkey; National Magnetic Resonance Research Center (UMRAM), Bilkent University, Ankara 06800, Turkey
| | - Salman U H Dar
- Department of Electrical and Electronics Engineering, Bilkent University, Ankara 06800, Turkey; National Magnetic Resonance Research Center (UMRAM), Bilkent University, Ankara 06800, Turkey
| | - Tolga Çukur
- Department of Electrical and Electronics Engineering, Bilkent University, Ankara 06800, Turkey; National Magnetic Resonance Research Center (UMRAM), Bilkent University, Ankara 06800, Turkey; Neuroscience Program, Bilkent University, Ankara 06800, Turkey.
| |
Collapse
|
40
|
Wu Z, Liao W, Yan C, Zhao M, Liu G, Ma N, Li X. Deep learning based MRI reconstruction with transformer. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2023; 233:107452. [PMID: 36924533 DOI: 10.1016/j.cmpb.2023.107452] [Citation(s) in RCA: 5] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/17/2022] [Revised: 02/19/2023] [Accepted: 02/27/2023] [Indexed: 06/18/2023]
Abstract
Magnetic resonance imaging (MRI) has become one of the most powerful imaging techniques in medical diagnosis, yet the prolonged scanning time becomes a bottleneck for application. Reconstruction methods based on compress sensing (CS) have made progress in reducing this cost by acquiring fewer points in the k-space. Traditional CS methods impose restrictions from different sparse domains to regularize the optimization that always requires balancing time with accuracy. Neural network techniques enable learning a better prior from sample pairs and generating the results in an analytic way. In this paper, we propose a deep learning based reconstruction method to restore high-quality MRI images from undersampled k-space data in an end-to-end style. Unlike prior literature adopting convolutional neural networks (CNN), advanced Swin Transformer is used as the backbone of our work, which proved to be powerful in extracting deep features of the image. In addition, we combined the k-space consistency in the output and further improved the quality. We compared our models with several reconstruction methods and variants, and the experiment results proved that our model achieves the best results in samples at low sampling rates. The source code of KTMR could be acquired at https://github.com/BITwzl/KTMR.
Collapse
Affiliation(s)
- Zhengliang Wu
- School of Computer Science & Technology, Beijing Institute of Technology, No. 5, South Street, Zhongguancun, Beijing, 100081, China.
| | - Weibin Liao
- School of Computer Science & Technology, Beijing Institute of Technology, No. 5, South Street, Zhongguancun, Beijing, 100081, China
| | - Chao Yan
- School of Computer Science & Technology, Beijing Institute of Technology, No. 5, South Street, Zhongguancun, Beijing, 100081, China
| | - Mangsuo Zhao
- Department of Neurology, Yuquan Hospital, School of Clinical Medicine, Tsinghua University, Beijing, 100039, China
| | - Guowen Liu
- Big Data and Engineering Research Center, Beijing Children's Hospital, Capital Medical University, Department of Echocardiography, Beijing, 100045, China; Beijing Advanced Innovation Center for Big Data-Based Precision Medicine, Beihang University & Capital Medical University, Beijing, 100083, China
| | - Ning Ma
- Big Data and Engineering Research Center, Beijing Children's Hospital, Capital Medical University, Department of Echocardiography, Beijing, 100045, China; Beijing Advanced Innovation Center for Big Data-Based Precision Medicine, Beihang University & Capital Medical University, Beijing, 100083, China.
| | - Xuesong Li
- School of Computer Science & Technology, Beijing Institute of Technology, No. 5, South Street, Zhongguancun, Beijing, 100081, China.
| |
Collapse
|
41
|
Zhou L, Zhu M, Xiong D, Ouyang L, Ouyang Y, Chen Z, Zhang X. RNLFNet: Residual non-local Fourier network for undersampled MRI reconstruction. Biomed Signal Process Control 2023. [DOI: 10.1016/j.bspc.2023.104632] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/08/2023]
|
42
|
Li J, Chen J, Tang Y, Wang C, Landman BA, Zhou SK. Transforming medical imaging with Transformers? A comparative review of key properties, current progresses, and future perspectives. Med Image Anal 2023; 85:102762. [PMID: 36738650 PMCID: PMC10010286 DOI: 10.1016/j.media.2023.102762] [Citation(s) in RCA: 22] [Impact Index Per Article: 22.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/02/2022] [Revised: 01/18/2023] [Accepted: 01/27/2023] [Indexed: 02/01/2023]
Abstract
Transformer, one of the latest technological advances of deep learning, has gained prevalence in natural language processing or computer vision. Since medical imaging bear some resemblance to computer vision, it is natural to inquire about the status quo of Transformers in medical imaging and ask the question: can the Transformer models transform medical imaging? In this paper, we attempt to make a response to the inquiry. After a brief introduction of the fundamentals of Transformers, especially in comparison with convolutional neural networks (CNNs), and highlighting key defining properties that characterize the Transformers, we offer a comprehensive review of the state-of-the-art Transformer-based approaches for medical imaging and exhibit current research progresses made in the areas of medical image segmentation, recognition, detection, registration, reconstruction, enhancement, etc. In particular, what distinguishes our review lies in its organization based on the Transformer's key defining properties, which are mostly derived from comparing the Transformer and CNN, and its type of architecture, which specifies the manner in which the Transformer and CNN are combined, all helping the readers to best understand the rationale behind the reviewed approaches. We conclude with discussions of future perspectives.
Collapse
Affiliation(s)
- Jun Li
- Key Lab of Intelligent Information Processing of Chinese Academy of Sciences (CAS), Institute of Computing Technology, CAS, Beijing 100190, China
| | - Junyu Chen
- Russell H. Morgan Department of Radiology and Radiological Science, Johns Hopkins Medical Institutes, Baltimore, MD, USA
| | - Yucheng Tang
- Department of Electrical and Computer Engineering, Vanderbilt University, Nashville, TN, USA
| | - Ce Wang
- Key Lab of Intelligent Information Processing of Chinese Academy of Sciences (CAS), Institute of Computing Technology, CAS, Beijing 100190, China
| | - Bennett A Landman
- Department of Electrical and Computer Engineering, Vanderbilt University, Nashville, TN, USA
| | - S Kevin Zhou
- Key Lab of Intelligent Information Processing of Chinese Academy of Sciences (CAS), Institute of Computing Technology, CAS, Beijing 100190, China; School of Biomedical Engineering & Suzhou Institute for Advanced Research, Center for Medical Imaging, Robotics, and Analytic Computing & Learning (MIRACLE), University of Science and Technology of China, Suzhou 215123, China.
| |
Collapse
|
43
|
Oscanoa JA, Middione MJ, Alkan C, Yurt M, Loecher M, Vasanawala SS, Ennis DB. Deep Learning-Based Reconstruction for Cardiac MRI: A Review. Bioengineering (Basel) 2023; 10:334. [PMID: 36978725 PMCID: PMC10044915 DOI: 10.3390/bioengineering10030334] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/02/2023] [Revised: 03/03/2023] [Accepted: 03/03/2023] [Indexed: 03/09/2023] Open
Abstract
Cardiac magnetic resonance (CMR) is an essential clinical tool for the assessment of cardiovascular disease. Deep learning (DL) has recently revolutionized the field through image reconstruction techniques that allow unprecedented data undersampling rates. These fast acquisitions have the potential to considerably impact the diagnosis and treatment of cardiovascular disease. Herein, we provide a comprehensive review of DL-based reconstruction methods for CMR. We place special emphasis on state-of-the-art unrolled networks, which are heavily based on a conventional image reconstruction framework. We review the main DL-based methods and connect them to the relevant conventional reconstruction theory. Next, we review several methods developed to tackle specific challenges that arise from the characteristics of CMR data. Then, we focus on DL-based methods developed for specific CMR applications, including flow imaging, late gadolinium enhancement, and quantitative tissue characterization. Finally, we discuss the pitfalls and future outlook of DL-based reconstructions in CMR, focusing on the robustness, interpretability, clinical deployment, and potential for new methods.
Collapse
Affiliation(s)
- Julio A. Oscanoa
- Department of Bioengineering, Stanford University, Stanford, CA 94305, USA
- Department of Radiology, Stanford University, Stanford, CA 94305, USA
| | | | - Cagan Alkan
- Department of Electrical Engineering, Stanford University, Stanford, CA 94305, USA
| | - Mahmut Yurt
- Department of Electrical Engineering, Stanford University, Stanford, CA 94305, USA
| | - Michael Loecher
- Department of Radiology, Stanford University, Stanford, CA 94305, USA
| | | | - Daniel B. Ennis
- Department of Radiology, Stanford University, Stanford, CA 94305, USA
| |
Collapse
|
44
|
Shen J, Lu S, Qu R, Zhao H, Zhang L, Chang A, Zhang Y, Fu W, Zhang Z. A boundary-guided transformer for measuring distance from rectal tumor to anal verge on magnetic resonance images. PATTERNS 2023; 4:100711. [PMID: 37123445 PMCID: PMC10140608 DOI: 10.1016/j.patter.2023.100711] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/15/2022] [Revised: 10/17/2022] [Accepted: 02/24/2023] [Indexed: 03/29/2023]
Abstract
Accurate measurement of the distance from the tumor's lowest boundary to the anal verge (DTAV) provides an important reference value for treatment of rectal cancer, but the standard measurement method (colonoscopy) causes substantial pain. Therefore, we propose a method for automatically measuring the DTAV on sagittal magnetic resonance (MR) images. We designed a boundary-guided transformer that can accurately segment the rectum and tumor. From the segmentation results, we estimated the DTAV by automatically extracting the anterior rectal wall from the tumor's lowest point to the anal verge and then calculating its physical length. Experiments were conducted on a rectal tumor MR imaging (MRI) dataset to evaluate the efficacy of our method. The results showed that our method outperformed surgeons with 6 years of experience (p < 0.001). Furthermore, by referring to our segmentation results, attending and resident surgeons could improve their measurement precision and efficiency.
Collapse
Affiliation(s)
- Jianjun Shen
- Department of Electronics, Tsinghua University, Beijing 100084, China
| | - Siyi Lu
- Department of General Surgery, Peking University Third Hospital, Beijing 100191, China
- Cancer Center, Peking University Third Hospital, Beijing 100191 China
| | - Ruize Qu
- Department of General Surgery, Peking University Third Hospital, Beijing 100191, China
- Cancer Center, Peking University Third Hospital, Beijing 100191 China
| | - Hao Zhao
- Intel Labs, Beijing 100190, China
| | - Li Zhang
- Department of Electronics, Tsinghua University, Beijing 100084, China
| | - An Chang
- Department of Electronics, Tsinghua University, Beijing 100084, China
| | - Yu Zhang
- School of Astronautics, Beihang University, Beijing 102206, China
- Corresponding author
| | - Wei Fu
- Department of General Surgery, Peking University Third Hospital, Beijing 100191, China
- Cancer Center, Peking University Third Hospital, Beijing 100191 China
- Corresponding author
| | - Zhipeng Zhang
- Department of General Surgery, Peking University Third Hospital, Beijing 100191, China
- Cancer Center, Peking University Third Hospital, Beijing 100191 China
- Corresponding author
| |
Collapse
|
45
|
Applegate MB, Kose K, Ghimire S, Rajadhyaksha M, Dy J. Self-supervised denoising of Nyquist-sampled volumetric images via deep learning. J Med Imaging (Bellingham) 2023; 10:024005. [PMID: 36992871 PMCID: PMC10042483 DOI: 10.1117/1.jmi.10.2.024005] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/27/2022] [Accepted: 03/06/2023] [Indexed: 03/29/2023] Open
Abstract
Purpose Deep learning has demonstrated excellent performance enhancing noisy or degraded biomedical images. However, many of these models require access to a noise-free version of the images to provide supervision during training, which limits their utility. Here, we develop an algorithm (noise2Nyquist) that leverages the fact that Nyquist sampling provides guarantees about the maximum difference between adjacent slices in a volumetric image, which allows denoising to be performed without access to clean images. We aim to show that our method is more broadly applicable and more effective than other self-supervised denoising algorithms on real biomedical images, and provides comparable performance to algorithms that need clean images during training. Approach We first provide a theoretical analysis of noise2Nyquist and an upper bound for denoising error based on sampling rate. We go on to demonstrate its effectiveness in denoising in a simulated example as well as real fluorescence confocal microscopy, computed tomography, and optical coherence tomography images. Results We find that our method has better denoising performance than existing self-supervised methods and is applicable to datasets where clean versions are not available. Our method resulted in peak signal to noise ratio (PSNR) within 1 dB and structural similarity (SSIM) index within 0.02 of supervised methods. On medical images, it outperforms existing self-supervised methods by an average of 3 dB in PSNR and 0.1 in SSIM. Conclusion noise2Nyquist can be used to denoise any volumetric dataset sampled at at least the Nyquist rate making it useful for a wide variety of existing datasets.
Collapse
Affiliation(s)
- Matthew B. Applegate
- Northeastern University, Department of Electrical and Computer Engineering, Boston, Massachusetts, United States
| | - Kivanc Kose
- Dermatology Service at Memorial Sloan Kettering Cancer Center, New York, United States
| | - Sandesh Ghimire
- Northeastern University, Department of Electrical and Computer Engineering, Boston, Massachusetts, United States
| | - Milind Rajadhyaksha
- Dermatology Service at Memorial Sloan Kettering Cancer Center, New York, United States
| | - Jennifer Dy
- Northeastern University, Department of Electrical and Computer Engineering, Boston, Massachusetts, United States
- Address all correspondence to Jennifer Dy,
| |
Collapse
|
46
|
Hossain A, Islam MT, Abdul Rahim SK, Rahman MA, Rahman T, Arshad H, Khandakar A, Ayari MA, Chowdhury MEH. A Lightweight Deep Learning Based Microwave Brain Image Network Model for Brain Tumor Classification Using Reconstructed Microwave Brain (RMB) Images. BIOSENSORS 2023; 13:bios13020238. [PMID: 36832004 PMCID: PMC9954219 DOI: 10.3390/bios13020238] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/21/2022] [Revised: 01/23/2023] [Accepted: 01/30/2023] [Indexed: 05/27/2023]
Abstract
Computerized brain tumor classification from the reconstructed microwave brain (RMB) images is important for the examination and observation of the development of brain disease. In this paper, an eight-layered lightweight classifier model called microwave brain image network (MBINet) using a self-organized operational neural network (Self-ONN) is proposed to classify the reconstructed microwave brain (RMB) images into six classes. Initially, an experimental antenna sensor-based microwave brain imaging (SMBI) system was implemented, and RMB images were collected to create an image dataset. It consists of a total of 1320 images: 300 images for the non-tumor, 215 images for each single malignant and benign tumor, 200 images for each double benign tumor and double malignant tumor, and 190 images for the single benign and single malignant tumor classes. Then, image resizing and normalization techniques were used for image preprocessing. Thereafter, augmentation techniques were applied to the dataset to make 13,200 training images per fold for 5-fold cross-validation. The MBINet model was trained and achieved accuracy, precision, recall, F1-score, and specificity of 96.97%, 96.93%, 96.85%, 96.83%, and 97.95%, respectively, for six-class classification using original RMB images. The MBINet model was compared with four Self-ONNs, two vanilla CNNs, ResNet50, ResNet101, and DenseNet201 pre-trained models, and showed better classification outcomes (almost 98%). Therefore, the MBINet model can be used for reliably classifying the tumor(s) using RMB images in the SMBI system.
Collapse
Affiliation(s)
- Amran Hossain
- Centre for Advanced Electronic and Communication Engineering, Department of Electrical, Electronic and Systems Engineering, Faculty of Engineering and Built Environment, Universiti Kebangsaan Malaysia, Bangi 43600, Malaysia
- Department of Computer Science and Engineering, Dhaka University of Engineering and Technology, Gazipur, Gazipur 1707, Bangladesh
| | - Mohammad Tariqul Islam
- Centre for Advanced Electronic and Communication Engineering, Department of Electrical, Electronic and Systems Engineering, Faculty of Engineering and Built Environment, Universiti Kebangsaan Malaysia, Bangi 43600, Malaysia
| | | | - Md Atiqur Rahman
- Centre for Advanced Electronic and Communication Engineering, Department of Electrical, Electronic and Systems Engineering, Faculty of Engineering and Built Environment, Universiti Kebangsaan Malaysia, Bangi 43600, Malaysia
| | - Tawsifur Rahman
- Department of Electrical Engineering, Qatar University, Doha 2713, Qatar
| | - Haslina Arshad
- Institute of IR4.0, Universiti Kebangsaan Malaysia (UKM), Bangi 43600, Malaysia
| | - Amit Khandakar
- Department of Electrical Engineering, Qatar University, Doha 2713, Qatar
| | - Mohamed Arslane Ayari
- Department of Civil and Architectural Engineering, Qatar University, Doha 2713, Qatar
| | | |
Collapse
|
47
|
A densely interconnected network for deep learning accelerated MRI. MAGMA (NEW YORK, N.Y.) 2023; 36:65-77. [PMID: 36103029 PMCID: PMC9992260 DOI: 10.1007/s10334-022-01041-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/18/2022] [Revised: 08/29/2022] [Accepted: 08/31/2022] [Indexed: 10/14/2022]
Abstract
OBJECTIVE To improve accelerated MRI reconstruction through a densely connected cascading deep learning reconstruction framework. MATERIALS AND METHODS A cascading deep learning reconstruction framework (reference model) was modified by applying three architectural modifications: input-level dense connections between cascade inputs and outputs, an improved deep learning sub-network, and long-range skip-connections between subsequent deep learning networks. An ablation study was performed, where five model configurations were trained on the NYU fastMRI neuro dataset with an end-to-end scheme conjunct on four- and eightfold acceleration. The trained models were evaluated by comparing their respective structural similarity index measure (SSIM), normalized mean square error (NMSE), and peak signal to noise ratio (PSNR). RESULTS The proposed densely interconnected residual cascading network (DIRCN), utilizing all three suggested modifications achieved a SSIM improvement of 8% and 11%, a NMSE improvement of 14% and 23%, and a PSNR improvement of 2% and 3% for four- and eightfold acceleration, respectively. In an ablation study, the individual architectural modifications all contributed to this improvement for both acceleration factors, by improving the SSIM, NMSE, and PSNR with approximately 2-4%, 4-9%, and 0.5-1%, respectively. CONCLUSION The proposed architectural modifications allow for simple adjustments on an already existing cascading framework to further improve the resulting reconstructions.
Collapse
|
48
|
Pombo G, Gray R, Cardoso MJ, Ourselin S, Rees G, Ashburner J, Nachev P. Equitable modelling of brain imaging by counterfactual augmentation with morphologically constrained 3D deep generative models. Med Image Anal 2023; 84:102723. [PMID: 36542907 PMCID: PMC10591114 DOI: 10.1016/j.media.2022.102723] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/07/2021] [Revised: 11/21/2022] [Accepted: 12/02/2022] [Indexed: 12/12/2022]
Abstract
We describe CounterSynth, a conditional generative model of diffeomorphic deformations that induce label-driven, biologically plausible changes in volumetric brain images. The model is intended to synthesise counterfactual training data augmentations for downstream discriminative modelling tasks where fidelity is limited by data imbalance, distributional instability, confounding, or underspecification, and exhibits inequitable performance across distinct subpopulations. Focusing on demographic attributes, we evaluate the quality of synthesised counterfactuals with voxel-based morphometry, classification and regression of the conditioning attributes, and the Fréchet inception distance. Examining downstream discriminative performance in the context of engineered demographic imbalance and confounding, we use UK Biobank and OASIS magnetic resonance imaging data to benchmark CounterSynth augmentation against current solutions to these problems. We achieve state-of-the-art improvements, both in overall fidelity and equity. The source code for CounterSynth is available at https://github.com/guilherme-pombo/CounterSynth.
Collapse
Affiliation(s)
- Guilherme Pombo
- UCL Queen Square Institute of Neurology, University College London, London, UK.
| | - Robert Gray
- UCL Queen Square Institute of Neurology, University College London, London, UK
| | - M Jorge Cardoso
- School of Biomedical Engineering & Imaging Sciences, King's College London, London, UK
| | - Sebastien Ourselin
- School of Biomedical Engineering & Imaging Sciences, King's College London, London, UK
| | - Geraint Rees
- UCL Queen Square Institute of Neurology, University College London, London, UK
| | - John Ashburner
- UCL Queen Square Institute of Neurology, University College London, London, UK
| | - Parashkev Nachev
- UCL Queen Square Institute of Neurology, University College London, London, UK
| |
Collapse
|
49
|
Zhao X, Yang T, Li B, Zhang X. SwinGAN: A dual-domain Swin Transformer-based generative adversarial network for MRI reconstruction. Comput Biol Med 2023; 153:106513. [PMID: 36603439 DOI: 10.1016/j.compbiomed.2022.106513] [Citation(s) in RCA: 11] [Impact Index Per Article: 11.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/01/2022] [Revised: 12/09/2022] [Accepted: 12/31/2022] [Indexed: 01/02/2023]
Abstract
Magnetic resonance imaging (MRI) is one of the most important modalities for clinical diagnosis. However, the main disadvantages of MRI are the long scanning time and the moving artifact caused by patient movement during prolonged imaging. It can also lead to patient anxiety and discomfort, so accelerated imaging is indispensable for MRI. Convolutional neural network (CNN) based methods have become the fact standard for medical image reconstruction, and generative adversarial network (GAN) have also been widely used. Nevertheless, due to the limited ability of CNN to capture long-distance information, it may lead to defects in the structure of the reconstructed images such as blurry contour. In this paper, we propose a novel Swin Transformer-based dual-domain generative adversarial network (SwinGAN) for accelerated MRI reconstruction. The SwinGAN consists of two generators: a frequency-domain generator and an image-domain generator. Both the generators utilize Swin Transformer as backbone for effectively capturing the long-distance dependencies. A contextual image relative position encoder (ciRPE) is designed to enhance the ability to capture local information. We extensively evaluate the method on the IXI brain dataset, MICCAI 2013 dataset and MRNet knee dataset. Compared with KIGAN, the peak signal-to-noise ratio (PSNR) and structural similarity index measure (SSIM) are improved by 6.1% and 1.49% to 37.64 dB and 0.98 on IXI dataset respectively, which demonstrates that our model can sufficiently utilize the local and global information of image. The model shows promising performance and robustness under different undersampling masks, different acceleration rates and different datasets. But it needs high hardware requirements with the increasing of the network parameters. The code is available at: https://github.com/learnerzx/SwinGAN.
Collapse
Affiliation(s)
- Xiang Zhao
- School of Information Science and Engineering, Henan University of Technology, Zhengzhou, 450001, China
| | - Tiejun Yang
- School of Artificial Intelligence and Big Data, Henan University of Technology, Zhengzhou, 450001, China; Key Laboratory of Grain Information Processing and Control (HAUT), Ministry of Education, Zhengzhou, China; Henan Key Laboratory of Grain Photoelectric Detection and Control (HAUT), Zhengzhou, Henan, China.
| | - Bingjie Li
- School of Information Science and Engineering, Henan University of Technology, Zhengzhou, 450001, China
| | - Xin Zhang
- School of Information Science and Engineering, Henan University of Technology, Zhengzhou, 450001, China
| |
Collapse
|
50
|
Yin L, Guo H, Zhang P, Li Y, Hui H, Du Y, Tian J. System matrix recovery based on deep image prior in magnetic particle imaging. Phys Med Biol 2023; 68. [PMID: 36584394 DOI: 10.1088/1361-6560/acaf47] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/22/2022] [Accepted: 12/30/2022] [Indexed: 12/31/2022]
Abstract
Objective.Magnetic particle imaging (MPI) is an emerging tomography imaging technique with high specificity and temporal-spatial resolution. MPI reconstruction based on the system matrix (SM) is an important research content in MPI. However, SM is usually obtained by measuring the response of an MPI scanner at all positions in the field of view. This process is very time-consuming, and the scanner will overheat in a long period of continuous operation, which is easy to generate thermal noise and affects MPI imaging performance.Approach.In this study, we propose a deep image prior-based method that prominently decreases the time of SM calibration. It is an unsupervised method that utilizes the neural network structure itself to recover a high-resolution SM from a downsampled SM without the need to train the network using a large amount of training data.Main results.Experiments on the Open MPI data show that the time of SM calibration can be greatly reduced with only slight degradation of image quality.Significance.This study provides a novel method for obtaining SM in MPI, which shows the potential to achieve SM recovery at a high downsampling rate. It is expected that this study will increase the practicability of MPI in biomedical applications and promote the development of MPI in the future.
Collapse
Affiliation(s)
- Lin Yin
- CAS Key Laboratory of Molecular Imaging, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, People's Republic of China.,Beijing Key Laboratory of Molecular Imaging, Beijing 100190, People's Republic of China.,University of Chinese Academy of Sciences, Beijing 100049, People's Republic of China
| | - Hongbo Guo
- School of Information Sciences and Technology, Northwest University, Xi'an, 710127, People's Republic of China
| | - Peng Zhang
- School of Computer and Information Technology, Beijing Jiaotong University, Beijing, 100044, People's Republic of China
| | - Yimeng Li
- Beijing Advanced Innovation Center for Big Data-Based Precision Medicine, School of Engineering Medicine, Beihang University, Beijing, 100191, People's Republic of China
| | - Hui Hui
- CAS Key Laboratory of Molecular Imaging, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, People's Republic of China.,Beijing Key Laboratory of Molecular Imaging, Beijing 100190, People's Republic of China.,University of Chinese Academy of Sciences, Beijing 100049, People's Republic of China
| | - Yang Du
- CAS Key Laboratory of Molecular Imaging, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, People's Republic of China.,Beijing Key Laboratory of Molecular Imaging, Beijing 100190, People's Republic of China.,University of Chinese Academy of Sciences, Beijing 100049, People's Republic of China
| | - Jie Tian
- CAS Key Laboratory of Molecular Imaging, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, People's Republic of China.,Beijing Key Laboratory of Molecular Imaging, Beijing 100190, People's Republic of China.,University of Chinese Academy of Sciences, Beijing 100049, People's Republic of China.,Beijing Advanced Innovation Center for Big Data-Based Precision Medicine, School of Engineering Medicine, Beihang University, Beijing, 100191, People's Republic of China
| |
Collapse
|