Image reconstruction by domain-transform manifold learning

Knowledge-driven deep learning for fast MR imaging: Undersampled MR image reconstruction from supervised to un-supervised learning

Deep learning (DL) has emerged as a leading approach in accelerating MRI. It employs deep neural networks to extract knowledge from available datasets and then applies the trained networks to reconstruct accurate images from limited measurements. Unlike natural image restoration problems, MRI involves physics-based imaging processes, unique data properties, and diverse imaging tasks. This domain knowledge needs to be integrated with data-driven approaches. Our review will introduce the significant challenges faced by such knowledge-driven DL approaches in the context of fast MRI along with several notable solutions, which include learning neural networks and addressing different imaging application scenarios. The traits and trends of these techniques have also been given which have shifted from supervised learning to semi-supervised learning, and finally, to unsupervised learning methods. In addition, MR vendors' choices of DL reconstruction have been provided along with some discussions on open questions and future directions, which are critical for the reliable imaging systems.

Integrating data distribution prior via Langevin dynamics for end-to-end MR reconstruction

PURPOSE

To develop a novel deep learning-based method inheriting the advantages of data distribution prior and end-to-end training for accelerating MRI.

METHODS

Langevin dynamics is used to formulate image reconstruction with data distribution before facilitate image reconstruction. The data distribution prior is learned implicitly through the end-to-end adversarial training to mitigate the hyper-parameter selection and shorten the testing time compared to traditional probabilistic reconstruction. By seamlessly integrating the deep equilibrium model, the iteration of Langevin dynamics culminates in convergence to a fix-point, ensuring the stability of the learned distribution.

RESULTS

The feasibility of the proposed method is evaluated on the brain and knee datasets. Retrospective results with uniform and random masks show that the proposed method demonstrates superior performance both quantitatively and qualitatively than the state-of-the-art.

CONCLUSION

The proposed method incorporating Langevin dynamics with end-to-end adversarial training facilitates efficient and robust reconstruction for MRI. Empirical evaluations conducted on brain and knee datasets compellingly demonstrate the superior performance of the proposed method in terms of artifact removing and detail preserving.

Improving quantitative MRI using self-supervised deep learning with model reinforcement: Demonstration for rapid T1 mapping

PURPOSE

This paper proposes a novel self-supervised learning framework that uses model reinforcement, REference-free LAtent map eXtraction with MOdel REinforcement (RELAX-MORE), for accelerated quantitative MRI (qMRI) reconstruction. The proposed method uses an optimization algorithm to unroll an iterative model-based qMRI reconstruction into a deep learning framework, enabling accelerated MR parameter maps that are highly accurate and robust.

METHODS

Unlike conventional deep learning methods which require large amounts of training data, RELAX-MORE is a subject-specific method that can be trained on single-subject data through self-supervised learning, making it accessible and practically applicable to many qMRI studies. Using quantitativeT 1 $$ {\mathrm{T}}_1 $$ mapping as an example, the proposed method was applied to the brain, knee and phantom data.

RESULTS

The proposed method generates high-quality MR parameter maps that correct for image artifacts, removes noise, and recovers image features in regions of imperfect image conditions. Compared with other state-of-the-art conventional and deep learning methods, RELAX-MORE significantly improves efficiency, accuracy, robustness, and generalizability for rapid MR parameter mapping.

CONCLUSION

This work demonstrates the feasibility of a new self-supervised learning method for rapid MR parameter mapping, that is readily adaptable to the clinical translation of qMRI.

Predictive uncertainty in deep learning-based MR image reconstruction using deep ensembles: Evaluation on the fastMRI data set

PURPOSE

To estimate pixel-wise predictive uncertainty for deep learning-based MR image reconstruction and to examine the impact of domain shifts and architecture robustness.

METHODS

Uncertainty prediction could provide a measure for robustness of deep learning (DL)-based MR image reconstruction from undersampled data. DL methods bear the risk of inducing reconstruction errors like in-painting of unrealistic structures or missing pathologies. These errors may be obscured by visual realism of DL reconstruction and thus remain undiscovered. Furthermore, most methods are task-agnostic and not well calibrated to domain shifts. We propose a strategy that estimates aleatoric (data) and epistemic (model) uncertainty, which entails training a deep ensemble (epistemic) with nonnegative log-likelihood (aleatoric) loss in addition to the conventional applied losses terms. The proposed procedure can be paired with any DL reconstruction, enabling investigations of their predictive uncertainties on a pixel level. Five different architectures were investigated on the fastMRI database. The impact on the examined uncertainty of in-distributional and out-of-distributional data with changes to undersampling pattern, imaging contrast, imaging orientation, anatomy, and pathology were explored.

RESULTS

Predictive uncertainty could be captured and showed good correlation to normalized mean squared error. Uncertainty was primarily focused along the aliased anatomies and on hyperintense and hypointense regions. The proposed uncertainty measure was able to detect disease prevalence shifts. Distinct predictive uncertainty patterns were observed for changing network architectures.

CONCLUSION

The proposed approach enables aleatoric and epistemic uncertainty prediction for DL-based MR reconstruction with an interpretable examination on a pixel level.

Low-dose CT reconstruction using dataset-free learning

Low-Dose computer tomography (LDCT) is an ideal alternative to reduce radiation risk in clinical applications. Although supervised-deep-learning-based reconstruction methods have demonstrated superior performance compared to conventional model-driven reconstruction algorithms, they require collecting massive pairs of low-dose and norm-dose CT images for neural network training, which limits their practical application in LDCT imaging. In this paper, we propose an unsupervised and training data-free learning reconstruction method for LDCT imaging that avoids the requirement for training data. The proposed method is a post-processing technique that aims to enhance the initial low-quality reconstruction results, and it reconstructs the high-quality images by neural work training that minimizes the ℓ1-norm distance between the CT measurements and their corresponding simulated sinogram data, as well as the total variation (TV) value of the reconstructed image. Moreover, the proposed method does not require to set the weights for both the data fidelity term and the plenty term. Experimental results on the AAPM challenge data and LoDoPab-CT data demonstrate that the proposed method is able to effectively suppress the noise and preserve the tiny structures. Also, these results demonstrate the rapid convergence and low computational cost of the proposed method. The source code is available at https://github.com/linfengyu77/IRLDCT.

The effects of back-projection variants in BPF-like TOF PET reconstruction using CNN filtration - Based on simulated and clinical brain data

BACKGROUND

The back-projection strategies such as confidence weighting (CW) and most likely annihilation position (MLAP) have been adopted into back-projection-and-filtering-like (BPF-like) deep reconstruction model and shown great potential on fast and accurate PET reconstruction. Although the two methods degenerate to an identical model at the time resolution of 0 ps, they represent two distinct approaches at the realistic time resolutions of current commercial systems. There is a lack of a systematic and fair assessment on these differences.

PURPOSE

This work aims to analyze the impact of back-projection variants on CNN-based PET image reconstruction to find the most effective back-projection model, and ultimately contribute to accurate PET reconstruction.

METHODS

Different back-projection strategies (CW and MLAP) and different angular view processing methods (view-summed and view-grouped) were considered, leading to the comparison of four back-projection variants integrated with the same CNN filtration model. Meanwhile, we investigated two strategies of physical effect compensation, either introducing pre-corrected data as the input or adding a channel of attenuation map to the CNN model. After training models separately on Monte-Carlo-simulated BrainWeb phantoms with full dose (events = 3×107), we tested them on both simulated phantoms and clinical brain scans with two dosage levels. For the performance assessment, peak signal-to-noise ratio (PSNR) and root mean square error (RMSE) were used to evaluate the pixel-wise error, structural similarity index (SSIM) to evaluate the structural similarity, and contrast recovery coefficient (CRC) in manually selected ROI to compare the region recovery.

RESULTS

Compared to two MLAP-based histo-image reconstruction models, two CW-based back-projected image methods produced clearer, sharper, and more detailed images, from both simulated and clinical data. For angular view processing methods, view-grouped histo-image improved image quality, while view-grouped cwbp-image showed no advantage except for contrast recovery. Quantitative analysis on simulated data demonstrated that the view-summed cwbp-image model achieved the best PSNR, RMSE, SSIM, while the 8-view cwbp-image model achieved the best CRC in lesions and the white matter. Additionally, the multi-channel input model including the back-projection image and attenuation map was proved to be the most efficient and simplest method for compensating for physical effects for brain data. Applying Gaussian blur to the histo-image yielded images with limited improvement. All above results hold for both the half-dose and the full-dose cases.

CONCLUSION

For brain imaging, the evaluation based on metrics PSNR, RMSE, SSIM, and CRC indicates that the view-summed CW-based back-projection variant is the most effective input for the BPF-like reconstruction model using CNN filtration, which can involve the attenuation map through an additional channel to effectively compensate for physical effects.

Fourier Domain Robust Denoising Decomposition and Adaptive Patch MRI Reconstruction

The sparsity of the Fourier transform domain has been applied to magnetic resonance imaging (MRI) reconstruction in k -space. Although unsupervised adaptive patch optimization methods have shown promise compared to data-driven-based supervised methods, the following challenges exist in MRI reconstruction: 1) in previous k -space MRI reconstruction tasks, MRI with noise interference in the acquisition process is rarely considered. 2) Differences in transform domains should be resolved to achieve the high-quality reconstruction of low undersampled MRI data. 3) Robust patch dictionary learning problems are usually nonconvex and NP-hard, and alternate minimization methods are often computationally expensive. In this article, we propose a method for Fourier domain robust denoising decomposition and adaptive patch MRI reconstruction (DDAPR). DDAPR is a two-step optimization method for MRI reconstruction in the presence of noise and low undersampled data. It includes the low-rank and sparse denoising reconstruction model (LSDRM) and the robust dictionary learning reconstruction model (RDLRM). In the first step, we propose LSDRM for different domains. For the optimization solution, the proximal gradient method is used to optimize LSDRM by singular value decomposition and soft threshold algorithms. In the second step, we propose RDLRM, which is an effective adaptive patch method by introducing a low-rank and sparse penalty adaptive patch dictionary and using a sparse rank-one matrix to approximate the undersampled data. Then, the block coordinate descent (BCD) method is used to optimize the variables. The BCD optimization process involves valid closed-form solutions. Extensive numerical experiments show that the proposed method has a better performance than previous methods in image reconstruction based on compressed sensing or deep learning.

Comparative analysis of image quality and interchangeability between standard and deep learning-reconstructed T2-weighted spine MRI

RATIONALE AND OBJECTIVES

MRI reconstruction of undersampled data using a deep learning (DL) network has been recently performed as part of accelerated imaging. Herein, we compared DL-reconstructed T2-weighted image (T2-WI) to conventional T2-WI regarding image quality and degenerative lesion detection.

MATERIALS AND METHODS

Sixty-two patients underwent C-spine (n = 27) or L-spine (n = 35) MRIs, including conventional and DL-reconstructed T2-WI. Image quality was assessed with non-uniformity measurement and 4-scale grading of structural visibility. Three readers (R1, R2, R3) independently assessed the presence and types of degenerative lesions. Student t-test was used to compare non-uniformity measurements. Interprotocol and interobserver agreement of structural visibility was analyzed with Wilcoxon signed-rank test and weighted-κ values, respectively. The diagnostic equivalence of degenerative lesion detection between two protocols was assessed with interchangeability test.

RESULTS

The acquisition time of DL-reconstructed images was reduced to about 21-58% compared to conventional images. Non-uniformity measurement was insignificantly different between the two images (p-value = 0.17). All readers rated DL-reconstructed images as showing the same or superior structural visibility compared to conventional images. Significantly improved visibility was observed at disk margin of C-spine (R1, p < 0.001; R2, p = 0.04) and dorsal root ganglia (R1, p = 0.03; R3, p = 0.02) and facet joint (R1, p = 0.04; R2, p < 0.001; R3, p = 0.03) of L-spine. Interobserver agreements of image quality were variable in each structure. Clinical interchangeability between two protocols for degenerative lesion detection was verified showing <5% in the upper bounds of 95% confidence intervals of agreement rate differences.

CONCLUSIONS

DL-reconstructed T2-WI demonstrates comparable image quality and diagnostic performance with conventional T2-WI in spine imaging, with reduced acquisition time.

A noise robust image reconstruction using slice aware cycle interpolator network for parallel imaging in MRI

BACKGROUND

Reducing Magnetic resonance imaging (MRI) scan time has been an important issue for clinical applications. In order to reduce MRI scan time, imaging acceleration was made possible by undersampling k-space data. This is achieved by leveraging additional spatial information from multiple, independent receiver coils, thereby reducing the number of sampled k-space lines.

PURPOSE

The aim of this study is to develop a deep-learning method for parallel imaging with a reduced number of auto-calibration signals (ACS) lines in noisy environments.

METHODS

A cycle interpolator network is developed for robust reconstruction of parallel MRI with a small number of ACS lines in noisy environments. The network estimates missing (unsampled) lines of each coil data, and these estimated missing lines are then utilized to re-estimate the sampled k-space lines. In addition, a slice aware reconstruction technique is developed for noise-robust reconstruction while reducing the number of ACS lines. We conducted an evaluation study using retrospectively subsampled data obtained from three healthy volunteers at 3T MRI, involving three different slice thicknesses (1.5, 3.0, and 4.5 mm) and three different image contrasts (T1w, T2w, and FLAIR).

RESULTS

Despite the challenges posed by substantial noise in cases with a limited number of ACS lines and thinner slices, the slice aware cycle interpolator network reconstructs the enhanced parallel images. It outperforms RAKI, effectively eliminating aliasing artifacts. Moreover, the proposed network outperforms GRAPPA and demonstrates the ability to successfully reconstruct brain images even under severe noisy conditions.

CONCLUSIONS

The slice aware cycle interpolator network has the potential to improve reconstruction accuracy for a reduced number of ACS lines in noisy environments.

Directional TV algorithm for image reconstruction from sparse-view projections in EPR imaging

Objective.Electron paramagnetic resonance (EPR) imaging is an advanced in vivo oxygen imaging modality. The main drawback of EPR imaging is the long scanning time. Sparse-view projections collection is an effective fast scanning pattern. However, the commonly-used filtered back projection (FBP) algorithm is not competent to accurately reconstruct images from sparse-view projections because of the severe streak artifacts. The aim of this work is to develop an advanced algorithm for sparse reconstruction of 3D EPR imaging.Methods.The optimization based algorithms including the total variation (TV) algorithm have proven to be effective in sparse reconstruction in EPR imaging. To further improve the reconstruction accuracy, we propose the directional TV (DTV) model and derive its Chambolle-Pock solving algorithm.Results.After the algorithm correctness validation on simulation data, we explore the sparse reconstruction capability of the DTV algorithm via a simulated six-sphere phantom and two real bottle phantoms filled with OX063 trityl solution and scanned by an EPR imager with a magnetic field strength of 250 G.Conclusion.Both the simulated and real data experiments show that the DTV algorithm is superior to the existing FBP and TV-type algorithms and a deep learning based method according to visual inspection and quantitative evaluations in sparse reconstruction of EPR imaging.Significance.These insights gained in this work may be used in the development of fast EPR imaging workflow of practical significance.

Inter-scanner super-resolution of 3D cine MRI using a transfer-learning network for MRgRT

Objective. Deep-learning networks for super-resolution (SR) reconstruction enhance the spatial-resolution of 3D magnetic resonance imaging (MRI) for MR-guided radiotherapy (MRgRT). However, variations between MRI scanners and patients impact the quality of SR for real-time 3D low-resolution (LR) cine MRI. In this study, we present a personalized super-resolution (psSR) network that incorporates transfer-learning to overcome the challenges in inter-scanner SR of 3D cine MRI.Approach: Development of the proposed psSR network comprises two-stages: (1) a cohort-specific SR (csSR) network using clinical patient datasets, and (2) a psSR network using transfer-learning to target datasets. The csSR network was developed by training on breath-hold and respiratory-gated high-resolution (HR) 3D MRIs and their k-space down-sampled LR MRIs from 53 thoracoabdominal patients scanned at 1.5 T. The psSR network was developed through transfer-learning to retrain the csSR network using a single breath-hold HR MRI and a corresponding 3D cine MRI from 5 healthy volunteers scanned at 0.55 T. Image quality was evaluated using the peak-signal-noise-ratio (PSNR) and the structure-similarity-index-measure (SSIM). The clinical feasibility was assessed by liver contouring on the psSR MRI using an auto-segmentation network and quantified using the dice-similarity-coefficient (DSC).Results. Mean PSNR and SSIM values of psSR MRIs were increased by 57.2% (13.8-21.7) and 94.7% (0.38-0.74) compared to cine MRIs, with the reference 0.55 T breath-hold HR MRI. In the contour evaluation, DSC was increased by 15% (0.79-0.91). Average time consumed for transfer-learning was 90 s, psSR was 4.51 ms per volume, and auto-segmentation was 210 ms, respectively.Significance. The proposed psSR reconstruction substantially increased image and segmentation quality of cine MRI in an average of 215 ms across the scanners and patients with less than 2 min of prerequisite transfer-learning. This approach would be effective in overcoming cohort- and scanner-dependency of deep-learning for MRgRT.

Mutli-modal straight flow matching for accelerated MR imaging

Diffusion models have garnered great interest lately in Magnetic Resonance (MR) image reconstruction. A key component of generating high-quality samples from noise is iterative denoising for thousands of steps. However, the complexity of inference steps has limited its applications. To solve the challenge in obtaining high-quality reconstructed images with fewer inference steps and computational complexity, we introduce a novel straight flow matching, based on a neural ordinary differential equation (ODE) generative model. Our model creates a linear path between undersampled images and reconstructed images, which can be accurately simulated with a few Euler steps. Furthermore, we propose a multi-modal straight flow matching model, which uses relatively easily available modalities as supplementary information to guide the reconstruction of target modalities. We introduce the low frequency fusion layer and the high frequency fusion layer into our multi-modal model, which has been proved to produce promising results in fusion tasks. The proposed multi-modal straight flow matching (MMSflow) achieves state-of-the-art performances in task of reconstruction in fastMRI and Brats-2020 and improves the sampling rate by an order of magnitude than other methods based on stochastic differential equations (SDE).

MRI reconstruction with enhanced self-similarity using graph convolutional network

BACKGROUND

Recent Convolutional Neural Networks (CNNs) perform low-error reconstruction in fast Magnetic Resonance Imaging (MRI). Most of them convolve the image with kernels and successfully explore the local information. Nonetheless, the non-local image information, which is embedded among image patches relatively far from each other, may be lost due to the limitation of the receptive field of the convolution kernel. We aim to incorporate a graph to represent non-local information and improve the reconstructed images by using the Graph Convolutional Enhanced Self-Similarity (GCESS) network.

METHODS

First, the image is reconstructed into the graph to extract the non-local self-similarity in the image. Second, GCESS uses spatial convolution and graph convolution to process the information in the image, so that local and non-local information can be effectively utilized. The network strengthens the non-local similarity between similar image patches while reconstructing images, making the reconstruction of structure more reliable.

RESULTS

Experimental results on in vivo knee and brain data demonstrate that the proposed method achieves better artifact suppression and detail preservation than state-of-the-art methods, both visually and quantitatively. Under 1D Cartesian sampling with 4 × acceleration (AF = 4), the PSNR of knee data reached 34.19 dB, 1.05 dB higher than that of the compared methods; the SSIM achieved 0.8994, 2% higher than the compared methods. Similar results were obtained for the reconstructed images under other sampling templates as demonstrated in our experiment.

CONCLUSIONS

The proposed method successfully constructs a hybrid graph convolution and spatial convolution network to reconstruct images. This method, through its training process, amplifies the non-local self-similarities, significantly benefiting the structural integrity of the reconstructed images. Experiments demonstrate that the proposed method outperforms the state-of-the-art reconstruction method in suppressing artifacts, as well as in preserving image details.

Accelerated MRI reconstructions via variational network and feature domain learning

We introduce three architecture modifications to enhance the performance of the end-to-end (E2E) variational network (VarNet) for undersampled MRI reconstructions. We first implemented the Feature VarNet, which propagates information throughout the cascades of the network in an N-channel feature-space instead of a 2-channel feature-space. Then, we add an attention layer that utilizes the spatial locations of Cartesian undersampling artifacts to further improve performance. Lastly, we combined the Feature and E2E VarNets into the Feature-Image (FI) VarNet, to facilitate cross-domain learning and boost accuracy. Reconstructions were evaluated on the fastMRI dataset using standard metrics and clinical scoring by three neuroradiologists. Feature and FI VarNets outperformed the E2E VarNet for 4 × , 5 × and 8 × Cartesian undersampling in all studied metrics. FI VarNet secured second place in the public fastMRI leaderboard for 4 × Cartesian undersampling, outperforming all open-source models in the leaderboard. Radiologists rated FI VarNet brain reconstructions with higher quality and sharpness than the E2E VarNet reconstructions. FI VarNet excelled in preserving anatomical details, including blood vessels, whereas E2E VarNet discarded or blurred them in some cases. The proposed FI VarNet enhances the reconstruction quality of undersampled MRI and could enable clinically acceptable reconstructions at higher acceleration factors than currently possible.

Whole-body magnetic resonance imaging at 0.05 Tesla

Despite a half-century of advancements, global magnetic resonance imaging (MRI) accessibility remains limited and uneven, hindering its full potential in health care. Initially, MRI development focused on low fields around 0.05 Tesla, but progress halted after the introduction of the 1.5 Tesla whole-body superconducting scanner in 1983. Using a permanent 0.05 Tesla magnet and deep learning for electromagnetic interference elimination, we developed a whole-body scanner that operates using a standard wall power outlet and without radiofrequency and magnetic shielding. We demonstrated its wide-ranging applicability for imaging various anatomical structures. Furthermore, we developed three-dimensional deep learning reconstruction to boost image quality by harnessing extensive high-field MRI data. These advances pave the way for affordable deep learning-powered ultra-low-field MRI scanners, addressing unmet clinical needs in diverse health care settings worldwide.

ReconU-Net: a direct PET image reconstruction using U-Net architecture with back projection-induced skip connection

Objective.This study aims to introduce a novel back projection-induced U-Net-shaped architecture, called ReconU-Net, based on the original U-Net architecture for deep learning-based direct positron emission tomography (PET) image reconstruction. Additionally, our objective is to visualize the behavior of direct PET image reconstruction by comparing the proposed ReconU-Net architecture with the original U-Net architecture and existing DeepPET encoder-decoder architecture without skip connections.Approach. The proposed ReconU-Net architecture uniquely integrates the physical model of the back projection operation into the skip connection. This distinctive feature facilitates the effective transfer of intrinsic spatial information from the input sinogram to the reconstructed image via an embedded physical model. The proposed ReconU-Net was trained using Monte Carlo simulation data from the Brainweb phantom and tested on both simulated and real Hoffman brain phantom data.Main results. The proposed ReconU-Net method provided better reconstructed image in terms of the peak signal-to-noise ratio and contrast recovery coefficient than the original U-Net and DeepPET methods. Further analysis shows that the proposed ReconU-Net architecture has the ability to transfer features of multiple resolutions, especially non-abstract high-resolution information, through skip connections. Unlike the U-Net and DeepPET methods, the proposed ReconU-Net successfully reconstructed the real Hoffman brain phantom, despite limited training on simulated data.Significance. The proposed ReconU-Net can improve the fidelity of direct PET image reconstruction, even with small training datasets, by leveraging the synergistic relationship between data-driven modeling and the physics model of the imaging process.

A cross-domain complex convolution neural network for undersampled magnetic resonance image reconstruction

To introduce a new cross-domain complex convolution neural network for accurate MR image reconstruction from undersampled k-space data. Most reconstruction methods utilize neural networks or cascade neural networks in either the image domain and/or the k-space domain. However, these methods encounter several challenges: 1) Applying neural networks directly in the k-space domain is suboptimal for feature extraction; 2) Classic image-domain networks have difficulty in fully extracting texture features; and 3) Existing cross-domain methods still face challenges in extracting and fusing features from both image and k-space domains simultaneously. In this work, we propose a novel deep-learning-based 2-D single-coil complex-valued MR reconstruction network termed TEID-Net. TEID-Net integrates three modules: 1) TE-Net, an image-domain-based sub-network designed to enhance contrast in input features by incorporating a Texture Enhancement Module; 2) ID-Net, an intermediate-domain sub-network tailored to operate in the image-Fourier space, with the specific goal of reducing aliasing artifacts realized by leveraging the superior incoherence property of the decoupled one-dimensional signals; and 3) TEID-Net, a cross-domain reconstruction network in which ID-Nets and TE-Nets are combined and cascaded to boost the quality of image reconstruction further. Extensive experiments have been conducted on the fastMRI and Calgary-Campinas datasets. Results demonstrate the effectiveness of the proposed TEID-Net in mitigating undersampling-induced artifacts and producing high-quality image reconstructions, outperforming several state-of-the-art methods while utilizing fewer network parameters. The cross-domain TEID-Net excels in restoring tissue structures and intricate texture details. The results illustrate that TEID-Net is particularly well-suited for regular Cartesian undersampling scenarios.

Informed Adaptive Sensing

For many inverse problems, the data on which the solution is based is acquired sequentially. We present an approach to the solution of such inverse problems where a sensor can be directed (or otherwise reconfigured on the fly) to acquire a particular measurement. An example problem is magnetic resonance image reconstruction. We use an estimate of mutual information derived from an empirical conditional distribution provided by a generative model to guide our measurement acquisition given measurements acquired so far. The conditionally generated data is a set of samples which are representative of the plausible solutions that satisfy the acquired measurements. We present experiments on toy and real world data sets. We focus on image data but we demonstrate that the method is applicable to a broader class of problems. We also show how a learned model such as a deep neural network can be leveraged to allow generalisation to unseen data. Our informed adaptive sensing method outperforms random sampling, variance based sampling, sparsity based methods, and compressed sensing.

3D cine-magnetic resonance imaging using spatial and temporal implicit neural representation learning (STINR-MR)

Objective. 3D cine-magnetic resonance imaging (cine-MRI) can capture images of the human body volume with high spatial and temporal resolutions to study anatomical dynamics. However, the reconstruction of 3D cine-MRI is challenged by highly under-sampled k-space data in each dynamic (cine) frame, due to the slow speed of MR signal acquisition. We proposed a machine learning-based framework, spatial and temporal implicit neural representation learning (STINR-MR), for accurate 3D cine-MRI reconstruction from highly under-sampled data.Approach. STINR-MR used a joint reconstruction and deformable registration approach to achieve a high acceleration factor for cine volumetric imaging. It addressed the ill-posed spatiotemporal reconstruction problem by solving a reference-frame 3D MR image and a corresponding motion model that deforms the reference frame to each cine frame. The reference-frame 3D MR image was reconstructed as a spatial implicit neural representation (INR) network, which learns the mapping from input 3D spatial coordinates to corresponding MR values. The dynamic motion model was constructed via a temporal INR, as well as basis deformation vector fields (DVFs) extracted from prior/onboard 4D-MRIs using principal component analysis. The learned temporal INR encodes input time points and outputs corresponding weighting factors to combine the basis DVFs into time-resolved motion fields that represent cine-frame-specific dynamics. STINR-MR was evaluated using MR data simulated from the 4D extended cardiac-torso (XCAT) digital phantom, as well as two MR datasets acquired clinically from human subjects. Its reconstruction accuracy was also compared with that of the model-based non-rigid motion estimation method (MR-MOTUS) and a deep learning-based method (TEMPEST).Main results. STINR-MR can reconstruct 3D cine-MR images with high temporal (<100 ms) and spatial (3 mm) resolutions. Compared with MR-MOTUS and TEMPEST, STINR-MR consistently reconstructed images with better image quality and fewer artifacts and achieved superior tumor localization accuracy via the solved dynamic DVFs. For the XCAT study, STINR reconstructed the tumors to a mean ± SD center-of-mass error of 0.9 ± 0.4 mm, compared to 3.4 ± 1.0 mm of the MR-MOTUS method. The high-frame-rate reconstruction capability of STINR-MR allows different irregular motion patterns to be accurately captured.Significance. STINR-MR provides a lightweight and efficient framework for accurate 3D cine-MRI reconstruction. It is a 'one-shot' method that does not require external data for pre-training, allowing it to avoid generalizability issues typically encountered in deep learning-based methods.

M2M-InvNet: Human Motor Cortex Mapping From Multi-Muscle Response Using TMS and Generative 3D Convolutional Network

Transcranial magnetic stimulation (TMS) is often applied to the motor cortex to stimulate a collection of motor evoked potentials (MEPs) in groups of peripheral muscles. The causal interface between TMS and MEP is the selective activation of neurons in the motor cortex; moving around the TMS 'spot' over the motor cortex causes different MEP responses. A question of interest is whether a collection of MEP responses can be used to identify the stimulated locations on the cortex, which could potentially be used to then place the TMS coil to produce chosen sets of MEPs. In this work we leverage our previous report on a 3D convolutional neural network (CNN) architecture that predicted MEPs from the induced electric field, to tackle an inverse imaging task in which we start with the MEPs and estimate the stimulated regions on the motor cortex. We present and evaluate five different inverse imaging CNN architectures, both conventional and generative, in terms of several measures of reconstruction accuracy. We found that one architecture, which we propose as M2M-InvNet, consistently achieved the best performance.

ICF-PR-Net: a deep phase retrieval neural network for X-ray phase contrast imaging of inertial confinement fusion capsules

X-ray phase contrast imaging (XPCI) has demonstrated capability to characterize inertial confinement fusion (ICF) capsules, and phase retrieval can reconstruct phase information from intensity images. This study introduces ICF-PR-Net, a novel deep learning-based phase retrieval method for ICF-XPCI. We numerically constructed datasets based on ICF capsule shape features, and proposed an object-image loss function to add image formation physics to network training. ICF-PR-Net outperformed traditional methods as it exhibited satisfactory robustness against strong noise and nonuniform background and was well-suited for ICF-XPCI's constrained experimental conditions and single exposure limit. Numerical and experimental results showed that ICF-PR-Net accurately retrieved the phase and absorption while maintaining retrieval quality in different situations. Overall, the ICF-PR-Net enables the diagnosis of the inner interface and electron density of capsules to address ignition-preventing problems, such as hydrodynamic instability growth.

Hybrid-supervised deep learning for domain transfer 3D protoacoustic image reconstruction

Objective. Protoacoustic imaging showed great promise in providing real-time 3D dose verification of proton therapy. However, the limited acquisition angle in protoacoustic imaging induces severe artifacts, which impairs its accuracy for dose verification. In this study, we developed a hybrid-supervised deep learning method for protoacoustic imaging to address the limited view issue.Approach. We proposed a Recon-Enhance two-stage deep learning method. In the Recon-stage, a transformer-based network was developed to reconstruct initial pressure maps from raw acoustic signals. The network is trained in a hybrid-supervised approach, where it is first trained using supervision by the iteratively reconstructed pressure map and then fine-tuned using transfer learning and self-supervision based on the data fidelity constraint. In the enhance-stage, a 3D U-net is applied to further enhance the image quality with supervision from the ground truth pressure map. The final protoacoustic images are then converted to dose for proton verification.Main results. The results evaluated on a dataset of 126 prostate cancer patients achieved an average root mean squared errors (RMSE) of 0.0292, and an average structural similarity index measure (SSIM) of 0.9618, out-performing related start-of-the-art methods. Qualitative results also demonstrated that our approach addressed the limit-view issue with more details reconstructed. Dose verification achieved an average RMSE of 0.018, and an average SSIM of 0.9891. Gamma index evaluation demonstrated a high agreement (94.7% and 95.7% for 1%/3 mm and 1%/5 mm) between the predicted and the ground truth dose maps. Notably, the processing time was reduced to 6 s, demonstrating its feasibility for online 3D dose verification for prostate proton therapy.Significance. Our study achieved start-of-the-art performance in the challenging task of direct reconstruction from radiofrequency signals, demonstrating the great promise of PA imaging as a highly efficient and accurate tool forinvivo3D proton dose verification to minimize the range uncertainties of proton therapy to improve its precision and outcomes.

Improved reconstruction for highly accelerated propeller diffusion 1.5 T clinical MRI

PURPOSE

Propeller fast-spin-echo diffusion magnetic resonance imaging (FSE-dMRI) is essential for the diagnosis of Cholesteatoma. However, at clinical 1.5 T MRI, its signal-to-noise ratio (SNR) remains relatively low. To gain sufficient SNR, signal averaging (number of excitations, NEX) is usually used with the cost of prolonged scan time. In this work, we leveraged the benefits of Locally Low Rank (LLR) constrained reconstruction to enhance the SNR. Furthermore, we enhanced both the speed and SNR by employing Convolutional Neural Networks (CNNs) for the accelerated PROPELLER FSE-dMRI on a 1.5 T clinical scanner.

METHODS

Residual U-Net (RU-Net) was found to be efficient for propeller FSE-dMRI data. It was trained to predict 2-NEX images obtained by Locally Low Rank (LLR) constrained reconstruction and used 1-NEX images obtained via simplified reconstruction as the inputs. The brain scans from healthy volunteers and patients with cholesteatoma were performed for model training and testing. The performance of trained networks was evaluated with normalized root-mean-square-error (NRMSE), structural similarity index measure (SSIM), and peak SNR (PSNR).

RESULTS

For 4 × under-sampled with 7 blades data, online reconstruction appears to provide suboptimal images-some small details are missing due to high noise interferences. Offline LLR enables suppression of noises and discovering some small structures. RU-Net demonstrated further improvement compared to LLR by increasing 18.87% of PSNR, 2.11% of SSIM, and reducing 53.84% of NRMSE. Moreover, RU-Net is about 1500 × faster than LLR (0.03 vs. 47.59 s/slice).

CONCLUSION

The LLR remarkably enhances the SNR compared to online reconstruction. Moreover, RU-Net improves propeller FSE-dMRI as reflected in PSNR, SSIM, and NRMSE. It requires only 1-NEX data, which allows a 2 × scan time reduction. In addition, its speed is approximately 1500 times faster than that of LLR-constrained reconstruction.

IMJENSE: Scan-Specific Implicit Representation for Joint Coil Sensitivity and Image Estimation in Parallel MRI

Parallel imaging is a commonly used technique to accelerate magnetic resonance imaging (MRI) data acquisition. Mathematically, parallel MRI reconstruction can be formulated as an inverse problem relating the sparsely sampled k-space measurements to the desired MRI image. Despite the success of many existing reconstruction algorithms, it remains a challenge to reliably reconstruct a high-quality image from highly reduced k-space measurements. Recently, implicit neural representation has emerged as a powerful paradigm to exploit the internal information and the physics of partially acquired data to generate the desired object. In this study, we introduced IMJENSE, a scan-specific implicit neural representation-based method for improving parallel MRI reconstruction. Specifically, the underlying MRI image and coil sensitivities were modeled as continuous functions of spatial coordinates, parameterized by neural networks and polynomials, respectively. The weights in the networks and coefficients in the polynomials were simultaneously learned directly from sparsely acquired k-space measurements, without fully sampled ground truth data for training. Benefiting from the powerful continuous representation and joint estimation of the MRI image and coil sensitivities, IMJENSE outperforms conventional image or k-space domain reconstruction algorithms. With extremely limited calibration data, IMJENSE is more stable than supervised calibrationless and calibration-based deep-learning methods. Results show that IMJENSE robustly reconstructs the images acquired at 5× and 6× accelerations with only 4 or 8 calibration lines in 2D Cartesian acquisitions, corresponding to 22.0% and 19.5% undersampling rates. The high-quality results and scanning specificity make the proposed method hold the potential for further accelerating the data acquisition of parallel MRI.

SWISTA-Nets: Subband-adaptive wavelet iterative shrinkage thresholding networks for image reconstruction

Robust and interpretable image reconstruction is central to imageology applications in clinical practice. Prevalent deep networks, with strong learning ability to extract implicit information from data manifold, are still lack of prior knowledge introduced from mathematics or physics, leading to instability, poor structure interpretability and high computation cost. As to this issue, we propose two prior knowledge-driven networks to combine the good interpretability of mathematical methods and the powerful learnability of deep learning methods. Incorporating different kinds of prior knowledge, we propose subband-adaptive wavelet iterative shrinkage thresholding networks (SWISTA-Nets), where almost every network module is in one-to-one correspondence with each step involved in the iterative algorithm. By end-to-end training of proposed SWISTA-Nets, implicit information can be extracted from training data and guide the tuning process of key parameters that possess mathematical definition. The inverse problems associated with two medical imaging modalities, i.e., electromagnetic tomography and X-ray computational tomography are applied to validate the proposed networks. Both visual and quantitative results indicate that the SWISTA-Nets outperform mathematical methods and state-of-the-art prior knowledge-driven networks, especially with fewer training parameters, interpretable network structures and well robustness. We assume that our analysis will support further investigation of prior knowledge-driven networks in the field of ill-posed image reconstruction.

Data-driven normative values based on generative manifold learning for quantitative MRI

In medicine, abnormalities in quantitative metrics such as the volume reduction of one brain region of an individual versus a control group are often provided as deviations from so-called normal values. These normative reference values are traditionally calculated based on the quantitative values from a control group, which can be adjusted for relevant clinical co-variables, such as age or sex. However, these average normative values do not take into account the globality of the available quantitative information. For example, quantitative analysis of T1-weighted magnetic resonance images based on anatomical structure segmentation frequently includes over 100 cerebral structures in the quantitative reports, and these tend to be analyzed separately. In this study, we propose a global approach to personalized normative values for each brain structure using an unsupervised Artificial Intelligence technique known as generative manifold learning. We test the potential benefit of these personalized normative values in comparison with the more traditional average normative values on a population of patients with drug-resistant epilepsy operated for focal cortical dysplasia, as well as on a supplementary healthy group and on patients with Alzheimer's disease.

Breaking boundaries in radiology: redefining AI diagnostics via raw data ahead of reconstruction

Objective.In the realm of utilizing artificial intelligence (AI) for medical image analysis, the paradigm of 'signal-image-knowledge' has remained unchanged. However, the process of 'signal to image' inevitably introduces information distortion, ultimately leading to irrecoverable biases in the 'image to knowledge' process. Our goal is to skip reconstruction and build a diagnostic model directly from the raw data (signal).Approach. This study focuses on computed tomography (CT) and its raw data (sinogram) as the research subjects. We simulate the real-world process of 'human-signal-image' using the workflow 'CT-simulated data- reconstructed CT,' and we develop a novel AI predictive model directly targeting raw data (RCTM). This model comprises orientation, spatial, and global analysis modules, embodying the fusion of local to global information extraction from raw data. We selected 1994 patients with retrospective cases of solid lung nodules and modeled different types of data.Main results. We employed predefined radiomic features to assess the diagnostic feature differences caused by reconstruction. The results indicated that approximately 14% of the features had Spearman correlation coefficients below 0.8. These findings suggest that despite the increasing maturity of CT reconstruction algorithms, they still introduce perturbations to diagnostic features. Moreover, our proposed RCTM achieved an area under the curve (AUC) of 0.863 in the diagnosis task, showcasing a comprehensive superiority over models constructed from secondary reconstructed CTs (0.840, 0.822, and 0.825). Additionally, the performance of RCTM closely resembled that of models constructed from original CT scans (0.868, 0.878, and 0.866).Significance. The diagnostic and therapeutic approach directly based on CT raw data can enhance the precision of AI models and the concept of 'signal-to-image' can be extended to other types of imaging. AI diagnostic models tailored to raw data offer the potential to disrupt the traditional paradigm of 'signal-image-knowledge', opening up new avenues for more accurate medical diagnostics.

Sequential deep learning image enhancement models improve diagnostic confidence, lesion detectability, and image reconstruction time in PET

BACKGROUND

Investigate the potential benefits of sequential deployment of two deep learning (DL) algorithms namely DL-Enhancement (DLE) and DL-based time-of-flight (ToF) (DLT). DLE aims to enhance the rapidly reconstructed ordered-subset-expectation-maximisation algorithm (OSEM) images towards block-sequential-regularised-expectation-maximisation (BSREM) images, whereas DLT aims to improve the quality of BSREM images reconstructed without ToF. As the algorithms differ in their purpose, sequential application may allow benefits from each to be combined. 20 FDG PET-CT scans were performed on a Discovery 710 (D710) and 20 on Discovery MI (DMI; both GE HealthCare). PET data was reconstructed using five combinations of algorithms:1. ToF-BSREM, 2. ToF-OSEM + DLE, 3. OSEM + DLE + DLT, 4. ToF-OSEM + DLE + DLT, 5. ToF-BSREM + DLT. To assess image noise, 30 mm-diameter spherical VOIs were drawn in both lung and liver to measure standard deviation of voxels within the volume. In a blind clinical reading, two experienced readers rated the images on a five-point Likert scale based on lesion detectability, diagnostic confidence, and image quality.

RESULTS

Applying DLE + DLT reduced noise whilst improving lesion detectability, diagnostic confidence, and image reconstruction time. ToF-OSEM + DLE + DLT reconstructions demonstrated an increase in lesion SUVmax of 28 ± 14% (average ± standard deviation) and 11 ± 5% for data acquired on the D710 and DMI, respectively. The same reconstruction scored highest in clinical readings for both lesion detectability and diagnostic confidence for D710.

CONCLUSIONS

The combination of DLE and DLT increased diagnostic confidence and lesion detectability compared to ToF-BSREM images. As DLE + DLT used input OSEM images, and because DL inferencing was fast, there was a significant decrease in overall reconstruction time. This could have applications to total body PET.

Deep learning-based algorithms for low-dose CT imaging: A review

The computed tomography (CT) technique is extensively employed as an imaging modality in clinical settings. The radiation dose of CT, however, is significantly high, thereby raising concerns regarding the potential radiation damage it may cause. The reduction of X-ray exposure dose in CT scanning may result in a significant decline in imaging quality, thereby elevating the risk of missed diagnosis and misdiagnosis. The reduction of CT radiation dose and acquisition of high-quality images to meet clinical diagnostic requirements have always been a critical research focus and challenge in the field of CT. Over the years, scholars have conducted extensive research on enhancing low-dose CT (LDCT) imaging algorithms, among which deep learning-based algorithms have demonstrated superior performance. In this review, we initially introduced the conventional algorithms for CT image reconstruction along with their respective advantages and disadvantages. Subsequently, we provided a detailed description of four aspects concerning the application of deep neural networks in LDCT imaging process: preprocessing in the projection domain, post-processing in the image domain, dual-domain processing imaging, and direct deep learning-based reconstruction (DLR). Furthermore, an analysis was conducted to evaluate the merits and demerits of each method. The commercial and clinical applications of the LDCT-DLR algorithm were also presented in an overview. Finally, we summarized the existing issues pertaining to LDCT-DLR and concluded the paper while outlining prospective trends for algorithmic advancement.

Domain transformation learning for MR image reconstruction from dual domain input

Medical images are acquired through diverse imaging systems, with each system employing specific image reconstruction techniques to transform sensor data into images. In MRI, sensor data (i.e., k-space data) is encoded in the frequency domain, and fully sampled k-space data is transformed into an image using the inverse Fourier Transform. However, in efforts to reduce acquisition time, k-space is often subsampled, necessitating a sophisticated image reconstruction method beyond a simple transform. The proposed approach addresses this challenge by training a model to learn domain transform, generating the final image directly from undersampled k-space input. Significantly, to improve the stability of reconstruction from randomly subsampled k-space data, folded images are incorporated as supplementary inputs in the dual-input ETER-net. Moreover, modifications are made to the formation of inputs for the bi-RNN stages to accommodate non-fixed k-space trajectories. Experimental validation, encompassing both regular and irregular sampling trajectories, validates the method's effectiveness. The results demonstrated superior performance, measured by PSNR, SSIM, and VIF, across acceleration factors of 4 and 8. In summary, the dual-input ETER-net emerges as an effective both regular and irregular sampling trajectories, and accommodating diverse acceleration factors.

Deep learning-based PET image denoising and reconstruction: a review

This review focuses on positron emission tomography (PET) imaging algorithms and traces the evolution of PET image reconstruction methods. First, we provide an overview of conventional PET image reconstruction methods from filtered backprojection through to recent iterative PET image reconstruction algorithms, and then review deep learning methods for PET data up to the latest innovations within three main categories. The first category involves post-processing methods for PET image denoising. The second category comprises direct image reconstruction methods that learn mappings from sinograms to the reconstructed images in an end-to-end manner. The third category comprises iterative reconstruction methods that combine conventional iterative image reconstruction with neural-network enhancement. We discuss future perspectives on PET imaging and deep learning technology.

A Deep Learning-Based Integrated Framework for Quality-Aware Undersampled Cine Cardiac MRI Reconstruction and Analysis

Cine cardiac magnetic resonance (CMR) imaging is considered the gold standard for cardiac function evaluation. However, cine CMR acquisition is inherently slow and in recent decades considerable effort has been put into accelerating scan times without compromising image quality or the accuracy of derived results. In this article, we present a fully-automated, quality-controlled integrated framework for reconstruction, segmentation and downstream analysis of undersampled cine CMR data. The framework produces high quality reconstructions and segmentations, leading to undersampling factors that are optimised on a scan-by-scan basis. This results in reduced scan times and automated analysis, enabling robust and accurate estimation of functional biomarkers. To demonstrate the feasibility of the proposed approach, we perform simulations of radial k-space acquisitions using in-vivo cine CMR data from 270 subjects from the UK Biobank (with synthetic phase) and in-vivo cine CMR data from 16 healthy subjects (with real phase). The results demonstrate that the optimal undersampling factor varies for different subjects by approximately 1 to 2 seconds per slice. We show that our method can produce quality-controlled images in a mean scan time reduced from 12 to 4 seconds per slice, and that image quality is sufficient to allow clinically relevant parameters to be automatically estimated to lie within 5% mean absolute difference.

Annihilation-Net: Learned annihilation relation for dynamic MR imaging

BACKGROUND

Deep learning methods driven by the low-rank regularization have achieved attractive performance in dynamic magnetic resonance (MR) imaging. The effectiveness of existing methods lies mainly in their ability to capture interframe relationships using network modules, which are lack interpretability.

PURPOSE

This study aims to design an interpretable methodology for modeling interframe relationships using convolutiona networks, namely Annihilation-Net and use it for accelerating dynamic MRI.

METHODS

Based on the equivalence between Hankel matrix product and convolution, we utilize convolutional networks to learn the null space transform for characterizing low-rankness. We employ low-rankness to represent interframe correlations in dynamic MR imaging, while combining with sparse constraints in the compressed sensing framework. The corresponding optimization problem is solved in an iterative form with the semi-quadratic splitting method (HQS). The iterative steps are unrolled into a network, dubbed Annihilation-Net. All the regularization parameters and null space transforms are set as learnable in the Annihilation-Net.

RESULTS

Experiments on the cardiac cine dataset show that the proposed model outperforms other competing methods both quantitatively and qualitatively. The training set and test set have 800 and 118 images, respectively.

CONCLUSIONS

The proposed Annihilation-Net improves the reconstruction quality of accelerated dynamic MRI with better interpretability.

A spatially adaptive regularization based three-dimensional reconstruction network for quantitative susceptibility mapping

Objective.Quantitative susceptibility mapping (QSM) is a new imaging technique for non-invasive characterization of the composition and microstructure ofin vivotissues, and it can be reconstructed from local field measurements by solving an ill-posed inverse problem. Even for deep learning networks, it is not an easy task to establish an accurate quantitative mapping between two physical quantities of different units, i.e. field shift in Hz and susceptibility value in ppm for QSM.Approach. In this paper, we propose a spatially adaptive regularization based three-dimensional reconstruction network SAQSM. A spatially adaptive module is specially designed and a set of them at different resolutions are inserted into the network decoder, playing a role of cross-modality based regularization constraint. Therefore, the exact information of both field and magnitude data is exploited to adjust the scale and shift of feature maps, and thus any information loss or deviation occurred in previous layers could be effectively corrected. The network encoding has a dynamic perceptual initialization, which enables the network to overcome receptive field intervals and also strengthens its ability to detect features of various sizes.Main results. Experimental results on the brain data of healthy volunteers, clinical hemorrhage and simulated phantom with calcification demonstrate that SAQSM can achieve more accurate reconstruction with less susceptibility artifacts, while perform well on the stability and generalization even for severe lesion areas.Significance. This proposed framework may provide a valuable paradigm to quantitative mapping or multimodal reconstruction.

Systematic Review on Learning-based Spectral CT

Spectral computed tomography (CT) has recently emerged as an advanced version of medical CT and significantly improves conventional (single-energy) CT. Spectral CT has two main forms: dual-energy computed tomography (DECT) and photon-counting computed tomography (PCCT), which offer image improvement, material decomposition, and feature quantification relative to conventional CT. However, the inherent challenges of spectral CT, evidenced by data and image artifacts, remain a bottleneck for clinical applications. To address these problems, machine learning techniques have been widely applied to spectral CT. In this review, we present the state-of-the-art data-driven techniques for spectral CT.

Riemannian Manifold-Based Feature Space and Corresponding Image Clustering Algorithms

Image feature representation is a key factor influencing the accuracy of clustering. Traditional point-based feature spaces represent spectral features of an image independently and introduce spatial relationships of pixels in the image domain to enhance the contextual information expression ability. Mapping-based feature spaces aim to preserve the structure information, but the complex computation and the unexplainability of image features have a great impact on their applications. To this end, we propose an explicit feature space called Riemannian manifold feature space (RMFS) to present the contextual information in a unified way. First, the Gaussian probability distribution function (pdf) is introduced to characterize the features of a pixel in its neighborhood system in the image domain. Then, the feature-related pdfs are mapped to a Riemannian manifold, which constructs the proposed RMFS. In RMFS, a point can express the complex contextual information of corresponding pixel in the image domain, and pixels representing the same object are linearly distributed. This gives us a chance to convert nonlinear image segmentation problems to linear computation. To verify the superiority of the expression ability of the proposed RMFS, a linear clustering algorithm and a fuzzy linear clustering algorithm are proposed. Experimental results show that the proposed RMFS-based algorithms outperform their counterparts in the spectral feature space and the RMFS-based ones without the linear distribution characteristics. This indicates that the RMFS can better express features of an image than spectral feature space, and the expressed features can be easily used to construct linear segmentation models.

Accelerating CEST imaging using a model-based deep neural network with synthetic training data

PURPOSE

To develop a model-based deep neural network for high-quality image reconstruction of undersampled multi-coil CEST data.

THEORY AND METHODS

Inspired by the variational network (VN), the CEST image reconstruction equation is unrolled into a deep neural network (CEST-VN) with a k-space data-sharing block that takes advantage of the inherent redundancy in adjacent CEST frames and 3D spatial-frequential convolution kernels that exploit correlations in the x-ω domain. Additionally, a new pipeline based on multiple-pool Bloch-McConnell simulations is devised to synthesize multi-coil CEST data from publicly available anatomical MRI data. The proposed network is trained on simulated data with a CEST-specific loss function that jointly measures the structural and CEST contrast. The performance of CEST-VN was evaluated on four healthy volunteers and five brain tumor patients using retrospectively or prospectively undersampled data with various acceleration factors, and then compared with other conventional and state-of-the-art reconstruction methods.

RESULTS

The proposed CEST-VN method generated high-quality CEST source images and amide proton transfer-weighted maps in healthy and brain tumor subjects, consistently outperforming GRAPPA, blind compressed sensing, and the original VN. With the acceleration factors increasing from 3 to 6, CEST-VN with the same hyperparameters yielded similar and accurate reconstruction without apparent loss of details or increase of artifacts. The ablation studies confirmed the effectiveness of the CEST-specific loss function and data-sharing block used.

CONCLUSIONS

The proposed CEST-VN method can offer high-quality CEST source images and amide proton transfer-weighted maps from highly undersampled multi-coil data by integrating the deep learning prior and multi-coil sensitivity encoding model.

Inverse radon transform with deep learning: an application in cardiac motion correction

Objective. This paper addresses performing inverse radon transform (IRT) with artificial neural network (ANN) or deep learning, simultaneously with cardiac motion correction (MC). The suggested application domain is cardiac image reconstruction in emission or transmission tomography where IRT is relevant. Our main contribution is in proposing an ANN architecture that is particularly suitable for this purpose.Approach. We validate our approach with two types of datasets. First, we use an abstract object that looks like a heart to simulate motion-blurred radon transform. With the known ground truth in hand, we then train our proposed ANN architecture and validate its effectiveness in MC. Second, we used human cardiac gated datasets for training and validation of our approach. The gating mechanism bins data over time using the electro-cardiogram (ECG) signals for cardiac motion correction.Main results. We have shown that trained ANNs can perform motion-corrected image reconstruction directly from a motion-corrupted sinogram. We have compared our model against two other known ANN-based approaches.Significance. Our method paves the way for eliminating any need for hardware gating in medical imaging.

A Systematic Review and Identification of the Challenges of Deep Learning Techniques for Undersampled Magnetic Resonance Image Reconstruction

Deep learning (DL) in magnetic resonance imaging (MRI) shows excellent performance in image reconstruction from undersampled k-space data. Artifact-free and high-quality MRI reconstruction is essential for ensuring accurate diagnosis, supporting clinical decision-making, enhancing patient safety, facilitating efficient workflows, and contributing to the validity of research studies and clinical trials. Recently, deep learning has demonstrated several advantages over conventional MRI reconstruction methods. Conventional methods rely on manual feature engineering to capture complex patterns and are usually computationally demanding due to their iterative nature. Conversely, DL methods use neural networks with hundreds of thousands of parameters and automatically learn relevant features and representations directly from the data. Nevertheless, there are some limitations to DL-based techniques concerning MRI reconstruction tasks, such as the need for large, labeled datasets, the possibility of overfitting, and the complexity of model training. Researchers are striving to develop DL models that are more efficient, adaptable, and capable of providing valuable information for medical practitioners. We provide a comprehensive overview of the current developments and clinical uses by focusing on state-of-the-art DL architectures and tools used in MRI reconstruction. This study has three objectives. Our main objective is to describe how various DL designs have changed over time and talk about cutting-edge tactics, including their advantages and disadvantages. Hence, data pre- and post-processing approaches are assessed using publicly available MRI datasets and source codes. Secondly, this work aims to provide an extensive overview of the ongoing research on transformers and deep convolutional neural networks for rapid MRI reconstruction. Thirdly, we discuss several network training strategies, like supervised, unsupervised, transfer learning, and federated learning for rapid and efficient MRI reconstruction. Consequently, this article provides significant resources for future improvement of MRI data pre-processing and fast image reconstruction.

Dual-Domain Reconstruction Network Incorporating Multi-Level Wavelet Transform and Recurrent Convolution for Sparse View Computed Tomography Imaging

Sparse view computed tomography (SVCT) aims to reduce the number of X-ray projection views required for reconstructing the cross-sectional image of an object. While SVCT significantly reduces X-ray radiation dose and speeds up scanning, insufficient projection data give rise to issues such as severe streak artifacts and blurring in reconstructed images, thereby impacting the diagnostic accuracy of CT detection. To address this challenge, a dual-domain reconstruction network incorporating multi-level wavelet transform and recurrent convolution is proposed in this paper. The dual-domain network is composed of a sinogram domain network (SDN) and an image domain network (IDN). Multi-level wavelet transform is employed in both IDN and SDN to decompose sinograms and CT images into distinct frequency components, which are then processed through separate network branches to recover detailed information within their respective frequency bands. To capture global textures, artifacts, and shallow features in sinograms and CT images, a recurrent convolution unit (RCU) based on convolutional long and short-term memory (Conv-LSTM) is designed, which can model their long-range dependencies through recurrent calculation. Additionally, a self-attention-based multi-level frequency feature normalization fusion (MFNF) block is proposed to assist in recovering high-frequency components by aggregating low-frequency components. Finally, an edge loss function based on the Laplacian of Gaussian (LoG) is designed as the regularization term for enhancing the recovery of high-frequency edge structures. The experimental results demonstrate the effectiveness of our approach in reducing artifacts and enhancing the reconstruction of intricate structural details across various sparse views and noise levels. Our method excels in both performance and robustness, as evidenced by its superior outcomes in numerous qualitative and quantitative assessments, surpassing contemporary state-of-the-art CNNs or Transformer-based reconstruction methods.

Data-driven system matrix manipulation enabling fast functional imaging and intra-image nonrigid motion correction in tomography

Tomographic imaging modalities are described by large system matrices. Sparse sampling and tissue motion degrade system matrix and image quality. Various existing techniques improve the image quality without correcting the system matrices. Here, we compress the system matrices to improve computational efficiency (e.g., 42 times) using singular value decomposition and fast Fourier transform. Enabled by the efficiency, we propose (1) fast sparsely sampling functional imaging by incorporating a densely sampled prior image into the system matrix, which maintains the critical linearity while mitigating artifacts and (2) intra-image nonrigid motion correction by incorporating the motion as subdomain translations into the system matrix and reconstructing the translations together with the image iteratively. We demonstrate the methods in 3D photoacoustic computed tomography with significantly improved image qualities and clarify their applicability to X-ray CT and MRI or other types of imperfections due to the similarities in system matrices.

M3S-Net: multi-modality multi-branch multi-self-attention network with structure-promoting loss for low-dose PET/CT enhancement

Objective.PET (Positron Emission Tomography) inherently involves radiotracer injections and long scanning time, which raises concerns about the risk of radiation exposure and patient comfort. Reductions in radiotracer dosage and acquisition time can lower the potential risk and improve patient comfort, respectively, but both will also reduce photon counts and hence degrade the image quality. Therefore, it is of interest to improve the quality of low-dose PET images.Approach.A supervised multi-modality deep learning model, named M3S-Net, was proposed to generate standard-dose PET images (60 s per bed position) from low-dose ones (10 s per bed position) and the corresponding CT images. Specifically, we designed a multi-branch convolutional neural network with multi-self-attention mechanisms, which first extracted features from PET and CT images in two separate branches and then fused the features to generate the final generated PET images. Moreover, a novel multi-modality structure-promoting term was proposed in the loss function to learn the anatomical information contained in CT images.Main results.We conducted extensive numerical experiments on real clinical data collected from local hospitals. Compared with state-of-the-art methods, the proposed M3S-Net not only achieved higher objective metrics and better generated tumors, but also performed better in preserving edges and suppressing noise and artifacts.Significance.The experimental results of quantitative metrics and qualitative displays demonstrate that the proposed M3S-Net can generate high-quality PET images from low-dose ones, which are competable to standard-dose PET images. This is valuable in reducing PET acquisition time and has potential applications in dynamic PET imaging.

McSTRA: A multi-branch cascaded swin transformer for point spread function-guided robust MRI reconstruction

Deep learning MRI reconstruction methods are often based on Convolutional neural network (CNN) models; however, they are limited in capturing global correlations among image features due to the intrinsic locality of the convolution operation. Conversely, the recent vision transformer models (ViT) are capable of capturing global correlations by applying self-attention operations on image patches. Nevertheless, the existing transformer models for MRI reconstruction rarely leverage the physics of MRI. In this paper, we propose a novel physics-based transformer model titled, the Multi-branch Cascaded Swin Transformers (McSTRA) for robust MRI reconstruction. McSTRA combines several interconnected MRI physics-related concepts with the Swin transformers: it exploits global MRI features via the shifted window self-attention mechanism; it extracts MRI features belonging to different spectral components via a multi-branch setup; it iterates between intermediate de-aliasing and data consistency via a cascaded network with intermediate loss computations; furthermore, we propose a point spread function-guided positional embedding generation mechanism for the Swin transformers which exploit the spread of the aliasing artifacts for effective reconstruction. With the combination of all these components, McSTRA outperforms the state-of-the-art methods while demonstrating robustness in adversarial conditions such as higher accelerations, noisy data, different undersampling protocols, out-of-distribution data, and abnormalities in anatomy.

Hierarchical neural architecture search with adaptive global-local feature learning for Magnetic Resonance Image reconstruction

Neural architecture search (NAS) has been introduced into the design of deep neural network architectures for Magnetic Resonance Imaging (MRI) reconstruction since NAS-based methods can acquire the complex network architecture automatically without professional designing experience and improve the model's generalization ability. However, current NAS-based MRI reconstruction methods suffer from a lack of efficient operators in the search space, which leads to challenges in effectively recovering high-frequency details. This limitation is primarily due to the prevalent use of convolution operators in the current search space, which struggle to capture both global and local features of MR images simultaneously, resulting in insufficient information utilization. To address this issue, a generative adversarial network (GAN) based model is proposed to reconstruct the MR image from under-sampled K-space data. Firstly, parameterized global and local feature learning modules at multiple scales are added into the search space to improve the capability of recovering high-frequency details. Secondly, to mitigate the increased search time caused by the augmented search space, a hierarchical NAS is designed to learn the global-local feature learning modules that enable the reconstruction network to learn global and local information of MR images at different scales adaptively. Thirdly, to reduce the number of network parameters and computational complexity, the standard operations in global-local feature learning modules are replaced with lightweight operations. Finally, experiments on several publicly available brain MRI image datasets evaluate the performance of the proposed method. Compared to the state-of-the-art MRI reconstruction methods, the proposed method yields better reconstruction results in terms of peak signal-to-noise ratio and structural similarity at a lower computational cost. Additionally, our reconstruction results are validated through a brain tumor classification task, affirming the practicability of the proposed method. Our code is available at https://github.com/wwHwo/HNASMRI.

Neutron penumbral image reconstruction with a convolution neural network using fast Fourier transform

In Inertial Confinement Fusion (ICF), the asymmetry of a hot spot is an important influence factor in implosion performance. Neutron penumbral imaging, which serves as an encoded-aperture imaging technique, is one of the most important diagnostic methods for detecting the shape of a hot spot. The detector image is a uniformly bright range surrounded by a penumbral area, which presents the strength distribution of hot spots. The present diagnostic modality employs an indirect imaging technique, necessitating the reconstruction process to be a pivotal aspect of the imaging protocol. The accuracy of imaging and the applicable range are significantly influenced by the reconstruction algorithm employed. We develop a neural network named Fast Fourier transform Neural Network (FFTNN) to reconstruct two-dimensional neutron emission images from the penumbral area of the detector images. The FFTNN architecture consists of 16 layers that include a FFT layer, convolution layer, fully connected layer, dropout layer, and reshape layer. Due to the limitations in experimental data, we propose a phenomenological method for describing hot spots to generate datasets for training neural networks. The reconstruction performance of the trained FFTNN is better than that of the traditional Wiener filtering and Lucy-Richardson algorithm on the simulated dataset, especially when the noise level is high as indicated by the evaluation metrics, such as mean squared error and structure similar index measure. This proposed neural network provides a new perspective, paving the way for integrating neutron imaging diagnosis into ICF.

Spatial and channel attention-based conditional Wasserstein GAN for direct and rapid image reconstruction in ultrasound computed tomography

Ultrasound computed tomography (USCT) is an emerging technology that offers a noninvasive and radiation-free imaging approach with high sensitivity, making it promising for the early detection and diagnosis of breast cancer. The speed-of-sound (SOS) parameter plays a crucial role in distinguishing between benign masses and breast cancer. However, traditional SOS reconstruction methods face challenges in achieving a balance between resolution and computational efficiency, which hinders their clinical applications due to high computational complexity and long reconstruction times. In this paper, we propose a novel and efficient approach for direct SOS image reconstruction based on an improved conditional generative adversarial network. The generator directly reconstructs SOS images from time-of-flight information, eliminating the need for intermediate steps. Residual spatial-channel attention blocks are integrated into the generator to adaptively determine the relevance of arrival time from the transducer pair corresponding to each pixel in the SOS image. An ablation study verified the effectiveness of this module. Qualitative and quantitative evaluation results on breast phantom datasets demonstrate that this method is capable of rapidly reconstructing high-quality SOS images, achieving better generation results and image quality. Therefore, we believe that the proposed algorithm represents a new direction in the research area of USCT SOS reconstruction.

AliasNet: Alias artefact suppression network for accelerated phase-encode MRI

Sparse reconstruction is an important aspect of MRI, helping to reduce acquisition time and improve spatial-temporal resolution. Popular methods are based mostly on compressed sensing (CS), which relies on the random sampling of k-space to produce incoherent (noise-like) artefacts. Due to hardware constraints, 1D Cartesian phase-encode under-sampling schemes are popular for 2D CS-MRI. However, 1D under-sampling limits 2D incoherence between measurements, yielding structured aliasing artefacts (ghosts) that may be difficult to remove assuming a 2D sparsity model. Reconstruction algorithms typically deploy direction-insensitive 2D regularisation for these direction-associated artefacts. Recognising that phase-encode artefacts can be separated into contiguous 1D signals, we develop two decoupling techniques that enable explicit 1D regularisation and leverage the excellent 1D incoherence characteristics. We also derive a combined 1D + 2D reconstruction technique that takes advantage of spatial relationships within the image. Experiments conducted on retrospectively under-sampled brain and knee data demonstrate that combination of the proposed 1D AliasNet modules with existing 2D deep learned (DL) recovery techniques leads to an improvement in image quality. We also find AliasNet enables a superior scaling of performance compared to increasing the size of the original 2D network layers. AliasNet therefore improves the regularisation of aliasing artefacts arising from phase-encode under-sampling, by tailoring the network architecture to account for their expected appearance. The proposed 1D + 2D approach is compatible with any existing 2D DL recovery technique deployed for this application.

Revealing hidden patterns in deep neural network feature space continuum via manifold learning

Deep neural networks (DNNs) extract thousands to millions of task-specific features during model training for inference and decision-making. While visualizing these features is critical for comprehending the learning process and improving the performance of the DNNs, existing visualization techniques work only for classification tasks. For regressions, the feature points lie on a high dimensional continuum having an inherently complex shape, making a meaningful visualization of the features intractable. Given that the majority of deep learning applications are regression-oriented, developing a conceptual framework and computational method to reliably visualize the regression features is of great significance. Here, we introduce a manifold discovery and analysis (MDA) method for DNN feature visualization, which involves learning the manifold topology associated with the output and target labels of a DNN. MDA leverages the acquired topological information to preserve the local geometry of the feature space manifold and provides insightful visualizations of the DNN features, highlighting the appropriateness, generalizability, and adversarial robustness of a DNN. The performance and advantages of the MDA approach compared to the existing methods are demonstrated in different deep learning applications.

A deep unrolled neural network for real-time MRI-guided brain intervention

Accurate navigation and targeting are critical for neurological interventions including biopsy and deep brain stimulation. Real-time image guidance further improves surgical planning and MRI is ideally suited for both pre- and intra-operative imaging. However, balancing spatial and temporal resolution is a major challenge for real-time interventional MRI (i-MRI). Here, we proposed a deep unrolled neural network, dubbed as LSFP-Net, for real-time i-MRI reconstruction. By integrating LSFP-Net and a custom-designed, MR-compatible interventional device into a 3 T MRI scanner, a real-time MRI-guided brain intervention system is proposed. The performance of the system was evaluated using phantom and cadaver studies. 2D/3D real-time i-MRI was achieved with temporal resolutions of 80/732.8 ms, latencies of 0.4/3.66 s including data communication, processing and reconstruction time, and in-plane spatial resolution of 1 × 1 mm2. The results demonstrated that the proposed method enables real-time monitoring of the remote-controlled brain intervention, and showed the potential to be readily integrated into diagnostic scanners for image-guided neurosurgery.