1
|
Wang S, Wu R, Jia S, Diakite A, Li C, Liu Q, Zheng H, Ying L. Knowledge-driven deep learning for fast MR imaging: Undersampled MR image reconstruction from supervised to un-supervised learning. Magn Reson Med 2024; 92:496-518. [PMID: 38624162 DOI: 10.1002/mrm.30105] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/03/2023] [Revised: 03/19/2024] [Accepted: 03/20/2024] [Indexed: 04/17/2024]
Abstract
Deep learning (DL) has emerged as a leading approach in accelerating MRI. It employs deep neural networks to extract knowledge from available datasets and then applies the trained networks to reconstruct accurate images from limited measurements. Unlike natural image restoration problems, MRI involves physics-based imaging processes, unique data properties, and diverse imaging tasks. This domain knowledge needs to be integrated with data-driven approaches. Our review will introduce the significant challenges faced by such knowledge-driven DL approaches in the context of fast MRI along with several notable solutions, which include learning neural networks and addressing different imaging application scenarios. The traits and trends of these techniques have also been given which have shifted from supervised learning to semi-supervised learning, and finally, to unsupervised learning methods. In addition, MR vendors' choices of DL reconstruction have been provided along with some discussions on open questions and future directions, which are critical for the reliable imaging systems.
Collapse
Affiliation(s)
- Shanshan Wang
- Paul C Lauterbur Research Center for Biomedical Imaging, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
| | - Ruoyou Wu
- Paul C Lauterbur Research Center for Biomedical Imaging, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
| | - Sen Jia
- Paul C Lauterbur Research Center for Biomedical Imaging, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
| | - Alou Diakite
- Paul C Lauterbur Research Center for Biomedical Imaging, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
- University of Chinese Academy of Sciences, Beijing, China
| | - Cheng Li
- Paul C Lauterbur Research Center for Biomedical Imaging, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
| | - Qiegen Liu
- Department of Electronic Information Engineering, Nanchang University, Nanchang, China
| | - Hairong Zheng
- Paul C Lauterbur Research Center for Biomedical Imaging, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
| | - Leslie Ying
- Department of Biomedical Engineering and Department of Electrical Engineering, The State University of New York, Buffalo, New York, USA
| |
Collapse
|
2
|
Dong S, Shewarega A, Chapiro J, Cai Z, Hyder F, Coman D, Duncan JS. High-resolution extracellular pH imaging of liver cancer with multiparametric MR using Deep Image Prior. NMR IN BIOMEDICINE 2024; 37:e5145. [PMID: 38488205 DOI: 10.1002/nbm.5145] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/06/2023] [Revised: 02/19/2024] [Accepted: 02/20/2024] [Indexed: 07/11/2024]
Abstract
Noninvasive extracellular pH (pHe) mapping with Biosensor Imaging of Redundant Deviation in Shifts (BIRDS) using MR spectroscopic imaging (MRSI) has been demonstrated on 3T clinical MR scanners at 8 × 8 × 10 mm3 spatial resolution and applied to study various liver cancer treatments. Although pHe imaging at higher resolution can be achieved by extending the acquisition time, a postprocessing method to increase the resolution is preferable, to minimize the duration spent by the subject in the MR scanner. In this work, we propose to improve the spatial resolution of pHe mapping with BIRDS by incorporating anatomical information in the form of multiparametric MRI and using an unsupervised deep-learning technique, Deep Image Prior (DIP). Specifically, we used high-resolution T 1 , T 2 , and diffusion-weighted imaging (DWI) MR images of rabbits with VX2 liver tumors as inputs to a U-Net architecture to provide anatomical information. U-Net parameters were optimized to minimize the difference between the output super-resolution image and the experimentally acquired low-resolution pHe image using the mean-absolute error. In this way, the super-resolution pHe image would be consistent with both anatomical MR images and the low-resolution pHe measurement from the scanner. The method was developed based on data from 49 rabbits implanted with VX2 liver tumors. For evaluation, we also acquired high-resolution pHe images from two rabbits, which were used as ground truth. The results indicate a good match between the spatial characteristics of the super-resolution images and the high-resolution ground truth, supported by the low pixelwise absolute error.
Collapse
Affiliation(s)
- Siyuan Dong
- Department of Electrical Engineering, Yale University, New Haven, Connecticut, USA
| | - Annabella Shewarega
- Department of Radiology and Biomedical Imaging, Yale University, New Haven, Connecticut, USA
| | - Julius Chapiro
- Department of Radiology and Biomedical Imaging, Yale University, New Haven, Connecticut, USA
| | - Zhuotong Cai
- Department of Biomedical Engineering, Yale University, New Haven, Connecticut, USA
- Institute of Artificial Intelligence and Robotics, Xi'an Jiaotong University, Xi'an, China
| | - Fahmeed Hyder
- Department of Radiology and Biomedical Imaging, Yale University, New Haven, Connecticut, USA
- Department of Biomedical Engineering, Yale University, New Haven, Connecticut, USA
| | - Daniel Coman
- Department of Radiology and Biomedical Imaging, Yale University, New Haven, Connecticut, USA
- Department of Biomedical Engineering, Yale University, New Haven, Connecticut, USA
| | - James S Duncan
- Department of Electrical Engineering, Yale University, New Haven, Connecticut, USA
- Department of Radiology and Biomedical Imaging, Yale University, New Haven, Connecticut, USA
- Department of Biomedical Engineering, Yale University, New Haven, Connecticut, USA
| |
Collapse
|
3
|
Qayyum A, Ilahi I, Shamshad F, Boussaid F, Bennamoun M, Qadir J. Untrained Neural Network Priors for Inverse Imaging Problems: A Survey. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2023; 45:6511-6536. [PMID: 36063506 DOI: 10.1109/tpami.2022.3204527] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/15/2023]
Abstract
In recent years, advancements in machine learning (ML) techniques, in particular, deep learning (DL) methods have gained a lot of momentum in solving inverse imaging problems, often surpassing the performance provided by hand-crafted approaches. Traditionally, analytical methods have been used to solve inverse imaging problems such as image restoration, inpainting, and superresolution. Unlike analytical methods for which the problem is explicitly defined and the domain knowledge is carefully engineered into the solution, DL models do not benefit from such prior knowledge and instead make use of large datasets to predict an unknown solution to the inverse problem. Recently, a new paradigm of training deep models using a single image, named untrained neural network prior (UNNP) has been proposed to solve a variety of inverse tasks, e.g., restoration and inpainting. Since then, many researchers have proposed various applications and variants of UNNP. In this paper, we present a comprehensive review of such studies and various UNNP applications for different tasks and highlight various open research problems which require further research.
Collapse
|
4
|
Aghabiglou A, Eksioglu EM. Projection-Based cascaded U-Net model for MR image reconstruction. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2021; 207:106151. [PMID: 34052771 DOI: 10.1016/j.cmpb.2021.106151] [Citation(s) in RCA: 12] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/19/2020] [Accepted: 04/29/2021] [Indexed: 06/12/2023]
Abstract
BACKGROUND AND OBJECTIVE Background and Objective: Recent studies in deep learning reveal that the U-Net stands out among the diverse set of deep models as an effective network structure, especially for imaging inverse problems. Initially, the U-Net model was developed to solve segmentation problems for biomedical images while using an annotated dataset. In this paper, we will study a novel application of the U-Net structure for the important inverse problem of MRI reconstruction. Deep networks are particularly efficient for the speed-up of the MR image reconstruction process by decreasing the data acquisition time, and they can significantly reduce the aliasing artifacts caused by the undersampling in the k-space. Our aim is to develop a novel and efficient cascaded U-Net framework for reconstructing MR images from undersampled k-space data. The new framework should have improved reconstruction performance when compared to competing methodologies. METHODS In this paper, a novel cascaded framework utilizing the U-Net as a sub-block is being proposed. The introduced U-Net cascade structure is applied to the magnetic resonance image reconstruction problem. The connection between the cascaded U-Nets is realized in the form of a recently developed projection-based updated data consistency layer. The novel structure is implemented in the PyTorch environment, which is one of the standards for deep learning implementations. The recently created fastMRI dataset which forms an important benchmark for MRI reconstruction is used for training and testing purposes. RESULTS We present simulation results comparing the novel method with a variety of competitive deep networks. The new cascaded U-Net structures PSNR performance stands on average 1.28 dB higher than the baseline U-Net. The improvement, when compared to the standard CNN, is on average 3.32 dB. CONCLUSIONS The proposed cascaded U-Net configuration results in an improved reconstruction performance when compared to the CNN, the cascaded CNN, and also the singular U-Net structures, where the singular U-Net forms the baseline reconstruction method from the fastMRI package. The use of the projection-based updated data consistency layer also leads to improved quantitative (including SSIM, PSNR, and NMSE results) and qualitative results when compared to the use of the conventional data consistency layer.
Collapse
Affiliation(s)
- Amir Aghabiglou
- Graduate School of Science, Engineering and Technology, Istanbul Technical University, Istanbul, Turkey.
| | - Ender M Eksioglu
- Electronics and Communication Engineering Department, Istanbul Technical University, Istanbul, Turkey.
| |
Collapse
|
5
|
Wang S, Xiao T, Liu Q, Zheng H. Deep learning for fast MR imaging: A review for learning reconstruction from incomplete k-space data. Biomed Signal Process Control 2021. [DOI: 10.1016/j.bspc.2021.102579] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/22/2022]
|
6
|
Radial Undersampling-Based Interpolation Scheme for Multislice CSMRI Reconstruction Techniques. BIOMED RESEARCH INTERNATIONAL 2021; 2021:6638588. [PMID: 33954189 PMCID: PMC8057880 DOI: 10.1155/2021/6638588] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 12/17/2020] [Accepted: 04/05/2021] [Indexed: 11/18/2022]
Abstract
Magnetic Resonance Imaging (MRI) is an important yet slow medical imaging modality. Compressed sensing (CS) theory has enabled to accelerate the MRI acquisition process using some nonlinear reconstruction techniques from even 10% of the Nyquist samples. In recent years, interpolated compressed sensing (iCS) has further reduced the scan time, as compared to CS, by exploiting the strong interslice correlation of multislice MRI. In this paper, an improved efficient interpolated compressed sensing (EiCS) technique is proposed using radial undersampling schemes. The proposed efficient interpolation technique uses three consecutive slices to estimate the missing samples of the central target slice from its two neighboring slices. Seven different evaluation metrics are used to analyze the performance of the proposed technique such as structural similarity index measurement (SSIM), feature similarity index measurement (FSIM), mean square error (MSE), peak signal to noise ratio (PSNR), correlation (CORR), sharpness index (SI), and perceptual image quality evaluator (PIQE) and compared with the latest interpolation techniques. The simulation results show that the proposed EiCS technique has improved image quality and performance using both golden angle and uniform angle radial sampling patterns, with an even lower sampling ratio and maximum information content and using a more practical sampling scheme.
Collapse
|
7
|
Hashimoto F, Ohba H, Ote K, Kakimoto A, Tsukada H, Ouchi Y. 4D deep image prior: dynamic PET image denoising using an unsupervised four-dimensional branch convolutional neural network. Phys Med Biol 2021; 66:015006. [PMID: 33227725 DOI: 10.1088/1361-6560/abcd1a] [Citation(s) in RCA: 27] [Impact Index Per Article: 9.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/11/2022]
Abstract
Although convolutional neural networks (CNNs) demonstrate the superior performance in denoising positron emission tomography (PET) images, a supervised training of the CNN requires a pair of large, high-quality PET image datasets. As an unsupervised learning method, a deep image prior (DIP) has recently been proposed; it can perform denoising with only the target image. In this study, we propose an innovative procedure for the DIP approach with a four-dimensional (4D) branch CNN architecture in end-to-end training to denoise dynamic PET images. Our proposed 4D CNN architecture can be applied to end-to-end dynamic PET image denoising by introducing a feature extractor and a reconstruction branch for each time frame of the dynamic PET image. In the proposed DIP method, it is not necessary to prepare high-quality and large patient-related PET images. Instead, a subject's own static PET image is used as additional information, dynamic PET images are treated as training labels, and denoised dynamic PET images are obtained from the CNN outputs. Both simulation with [18F]fluoro-2-deoxy-D-glucose (FDG) and preclinical data with [18F]FDG and [11C]raclopride were used to evaluate the proposed framework. The results showed that our 4D DIP framework quantitatively and qualitatively outperformed 3D DIP and other unsupervised denoising methods. The proposed 4D DIP framework thus provides a promising procedure for dynamic PET image denoising.
Collapse
Affiliation(s)
- Fumio Hashimoto
- Central Research Laboratory, Hamamatsu Photonics K. K., 5000 Hirakuchi, Hamakita-ku, Hamamatsu 434-8601, Japan
| | | | | | | | | | | |
Collapse
|
8
|
Wang M, Wei S, Liang J, Zhou Z, Qu Q, Shi J, Zhang X. TPSSI-Net: Fast and Enhanced Two-Path Iterative Network for 3D SAR Sparse Imaging. IEEE TRANSACTIONS ON IMAGE PROCESSING : A PUBLICATION OF THE IEEE SIGNAL PROCESSING SOCIETY 2021; 30:7317-7332. [PMID: 34415832 DOI: 10.1109/tip.2021.3104168] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/13/2023]
Abstract
The emerging field of combining compressed sensing (CS) and three-dimensional synthetic aperture radar (3D SAR) imaging has shown significant potential to reduce sampling rate and improve image quality. However, the conventional CS-driven algorithms are always limited by huge computational costs and non-trivial tuning of parameters. In this article, to address this problem, we propose a two-path iterative framework dubbed TPSSI-Net for 3D SAR sparse imaging. By mapping the AMP into a layer-fixed deep neural network, each layer of TPSSI-Net consists of four modules in cascade corresponding to four steps of the AMP optimization. Differently, the Onsager terms in TPSSI-Net are modified to be differentiable and scaled by learnable coefficients. Rather than manually choosing a sparsifying basis, a two-path convolutional neural network (CNN) is developed and embedded in TPSSI-Net for nonlinear sparse representation in the complex-valued domain. All parameters are layer-varied and optimized by end-to-end training based on a channel-wise loss function, bounding both symmetry constraint and measurement fidelity. Finally, extensive SAR imaging experiments, including simulations and real-measured tests, demonstrate the effectiveness and high efficiency of the proposed TPSSI-Net.
Collapse
|
9
|
Vujović S, Draganić A, Lakičević Žarić M, Orović I, Daković M, Beko M, Stanković S. Sparse Analyzer Tool for Biomedical Signals. SENSORS 2020; 20:s20092602. [PMID: 32370285 PMCID: PMC7248901 DOI: 10.3390/s20092602] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 03/30/2020] [Revised: 04/24/2020] [Accepted: 04/27/2020] [Indexed: 11/16/2022]
Abstract
The virtual (software) instrument with a statistical analyzer for testing algorithms for biomedical signals' recovery in compressive sensing (CS) scenario is presented. Various CS reconstruction algorithms are implemented with the aim to be applicable for different types of biomedical signals and different applications with under-sampled data. Incomplete sampling/sensing can be considered as a sort of signal damage, where missing data can occur as a result of noise or the incomplete signal acquisition procedure. Many approaches for recovering the missing signal parts have been developed, depending on the signal nature. Here, several approaches and their applications are presented for medical signals and images. The possibility to analyze results using different statistical parameters is provided, with the aim to choose the most suitable approach for a specific application. The instrument provides manifold possibilities such as fitting different parameters for the considered signal and testing the efficiency under different percentages of missing data. The reconstruction accuracy is measured by the mean square error (MSE) between original and reconstructed signal. Computational time is important from the aspect of power requirements, thus enabling the selection of a suitable algorithm. The instrument contains its own signal database, but there is also the possibility to load any external data for analysis.
Collapse
Affiliation(s)
- Stefan Vujović
- Faculty of Electrical Engineering, University of Montenegro, 81000 Podgorica, Montenegro; (S.V.); (M.L.Ž.); (I.O.); (M.D.); (S.S.)
| | - Andjela Draganić
- Faculty of Electrical Engineering, University of Montenegro, 81000 Podgorica, Montenegro; (S.V.); (M.L.Ž.); (I.O.); (M.D.); (S.S.)
- Correspondence:
| | - Maja Lakičević Žarić
- Faculty of Electrical Engineering, University of Montenegro, 81000 Podgorica, Montenegro; (S.V.); (M.L.Ž.); (I.O.); (M.D.); (S.S.)
| | - Irena Orović
- Faculty of Electrical Engineering, University of Montenegro, 81000 Podgorica, Montenegro; (S.V.); (M.L.Ž.); (I.O.); (M.D.); (S.S.)
| | - Miloš Daković
- Faculty of Electrical Engineering, University of Montenegro, 81000 Podgorica, Montenegro; (S.V.); (M.L.Ž.); (I.O.); (M.D.); (S.S.)
| | - Marko Beko
- COPELABS, Universidade Lusófona de Humanidades e Tecnologias, 1700-097 Lisboa, Portugal;
- UNINOVA, Faculdade de Ciências e Tecnologia, 2829-517 Monte Caparica, Portugal
| | - Srdjan Stanković
- Faculty of Electrical Engineering, University of Montenegro, 81000 Podgorica, Montenegro; (S.V.); (M.L.Ž.); (I.O.); (M.D.); (S.S.)
| |
Collapse
|
10
|
Compressed-Sensing Magnetic Resonance Image Reconstruction Using an Iterative Convolutional Neural Network Approach. APPLIED SCIENCES-BASEL 2020. [DOI: 10.3390/app10061902] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
Abstract
Convolutional neural networks (CNNs) demonstrate excellent performance when employed to reconstruct the images obtained by compressed-sensing magnetic resonance imaging (CS-MRI). Our study aimed to enhance image quality by developing a novel iterative reconstruction approach that utilizes image-based CNNs and k-space correction to preserve original k-space data. In the proposed method, CNNs represent a priori information concerning image spaces. First, the CNNs are trained to map zero-filling images onto corresponding full-sampled images. Then, they recover the zero-filled part of the k-space data. Subsequently, k-space corrections, which involve the replacement of unfilled regions by original k-space data, are implemented to preserve the original k-space data. The above-mentioned processes are used iteratively. The performance of the proposed method was validated using a T2-weighted brain-image dataset, and experiments were conducted with several sampling masks. Finally, the proposed method was compared with other noniterative approaches to demonstrate its effectiveness. The aliasing artifacts in the reconstructed images obtained using the proposed approach were reduced compared to those using other state-of-the-art techniques. In addition, the quantitative results obtained in the form of the peak signal-to-noise ratio and structural similarity index demonstrated the effectiveness of the proposed method. The proposed CS-MRI method enhanced MR image quality with high-throughput examinations.
Collapse
|