Adaptive channel-modulated personalized federated learning for magnetic resonance image reconstruction

Magnetic resonance imaging (MRI) is extensively utilized in clinical practice for diagnostic purposes, owing to its non-invasive nature and remarkable ability to provide detailed characterization of soft tissues. However, its drawback lies in the prolonged scanning time. To accelerate MR imaging, how to reconstruct MR images from under-sampled data quickly and accurately has drawn intensive research interest; it, however, remains a challenging task. While some deep learning models have achieved promising performance in MRI reconstruction, these models usually require a substantial quantity of paired data for training, which proves challenging to gather and share owing to high scanning costs and data privacy concerns. Federated learning (FL) is a potential tool to alleviate these difficulties. It enables multiple clinical clients to collaboratively train a global model without compromising privacy. However, it is extremely challenging to fit a single model to diverse data distributions of different clients. Moreover, existing FL algorithms treat the features of each channel equally, lacking discriminative learning ability across feature channels, and hence hindering their representational capability. In this study, we propose a novel Adaptive Channel-Modulated Federal learning framework for personalized MRI reconstruction, dubbed as ACM-FedMRI. Specifically, considering each local client may focus on features in different channels, we first design a client-specific hypernetwork to guide the channel selection operation in order to optimize the extracted features. Additionally, we introduce a performance-based channel decoupling scheme, which dynamically separates the global model at the channel level to facilitate personalized adjustments based on the performance of individual clients. This approach eliminates the need for heuristic design of specific personalization layers. Extensive experiments on four datasets under two different settings show that our ACM-FedMRI achieves outstanding results compared to other cutting-edge federated learning techniques in the field of MRI reconstruction.

A Coarse-to-Fine Fusion Network for Small Liver Tumor Detection and Segmentation: A Real-World Study

Liver tumor semantic segmentation is a crucial task in medical image analysis that requires multiple MRI modalities. This paper proposes a novel coarse-to-fine fusion segmentation approach to detect and segment small liver tumors of various sizes. To enhance the segmentation accuracy of small liver tumors, the method incorporates a detection module and a CSR (convolution-SE-residual) module, which includes a convolution block, an SE (squeeze and excitation) module, and a residual module for fine segmentation. The proposed method demonstrates superior performance compared to conventional single-stage end-to-end networks. A private liver MRI dataset comprising 218 patients with a total of 3605 tumors, including 3273 tumors smaller than 3.0 cm, were collected for the proposed method. There are five types of liver tumors identified in this dataset: hepatocellular carcinoma (HCC); metastases of the liver; cholangiocarcinoma (ICC); hepatic cyst; and liver hemangioma. The results indicate that the proposed method outperforms the single segmentation networks 3D UNet and nnU-Net as well as the fusion networks of 3D UNet and nnU-Net with nnDetection. The proposed architecture was evaluated on a test set of 44 images, with an average Dice similarity coefficient (DSC) and recall of 86.9% and 86.7%, respectively, which is a 1% improvement compared to the comparison method. More importantly, compared to existing methods, our proposed approach demonstrates state-of-the-art performance in segmenting small objects with sizes smaller than 10 mm, achieving a Dice score of 85.3% and a malignancy detection rate of 87.5%.

Region-focused multi-view transformer-based generative adversarial network for cardiac cine MRI reconstruction

Cardiac cine magnetic resonance imaging (MRI) reconstruction is challenging due to spatial and temporal resolution trade-offs. Temporal correlation in cardiac cine MRI is informative and vital for understanding cardiac dynamic motion. Exploiting the temporal correlations in cine reconstruction is crucial to resolve aliasing artifacts and maintaining the cardiac motion patterns. However, existing methods have the following shortcomings: (1) they simultaneously compute pairwise correlations along spatial and temporal dimensions to establish dependencies, ignoring that learning spatial contextual information first will benefit the temporal modeling. (2) most studies neglect to focus on reconstructing the local cardiac regions, resulting in insufficient reconstruction accuracy due to a relatively large field of view. To address these problems, we propose a region-focused multi-view transformer-based generative adversarial network for cardiac cine MRI reconstruction. The proposed transformer divides consecutive cardiac frames into multiple views for cross-view feature extraction, establishing long-distance dependencies among features and effectively learning the spatio-temporal information. We further design a cross-view attention for spatio-temporal information fusion, ensuring the interaction of different spatio-temporal information in each view and capturing more temporal correlations of the cardiac motion. In addition, we introduce a cardiac region detection loss for improving the reconstruction quality of the cardiac region. Experimental results demonstrated that our method outperforms state-of-the-art methods. Especially with an acceleration factor as high as 10×, our method can reconstruct images with better accuracy and perceptual quality.

Multi-channel GAN-based calibration-free diffusion-weighted liver imaging with simultaneous coil sensitivity estimation and reconstruction

Introduction

Diffusion-weighted imaging (DWI) with parallel reconstruction may suffer from a mismatch between the coil calibration scan and imaging scan due to motions, especially for abdominal imaging.

Methods

This study aimed to construct an iterative multichannel generative adversarial network (iMCGAN)-based framework for simultaneous sensitivity map estimation and calibration-free image reconstruction. The study included 106 healthy volunteers and 10 patients with tumors.

Results

The performance of iMCGAN was evaluated in healthy participants and patients and compared with the SAKE, ALOHA-net, and DeepcomplexMRI reconstructions. The peak signal-to-noise ratio (PSNR), structural similarity index measure (SSIM), root mean squared error (RMSE), and histograms of apparent diffusion coefficient (ADC) maps were calculated for assessing image qualities. The proposed iMCGAN outperformed the other methods in terms of the PSNR (iMCGAN: 41.82 ± 2.14; SAKE: 17.38 ± 1.78; ALOHA-net: 20.43 ± 2.11 and DeepcomplexMRI: 39.78 ± 2.78) for b = 800 DWI with an acceleration factor of 4. Besides, the ghosting artifacts in the SENSE due to the mismatch between the DW image and the sensitivity maps were avoided using the iMCGAN model.

Discussion

The current model iteratively refined the sensitivity maps and the reconstructed images without additional acquisitions. Thus, the quality of the reconstructed image was improved, and the aliasing artifact was alleviated when motions occurred during the imaging procedure.

Added Value of Viscoelasticity for MRI-Based Prediction of Ki-67 Expression of Hepatocellular Carcinoma Using a Deep Learning Combined Radiomics (DLCR) Model

This study aimed to explore the added value of viscoelasticity measured by magnetic resonance elastography (MRE) in the prediction of Ki-67 expression in hepatocellular carcinoma (HCC) using a deep learning combined radiomics (DLCR) model. This retrospective study included 108 histopathology-proven HCC patients (93 males; age, 59.6 ± 11.0 years) who underwent preoperative MRI and MR elastography. They were divided into training (n = 87; 61.0 ± 9.8 years) and testing (n = 21; 60.6 ± 10.1 years) cohorts. An independent validation cohort including 43 patients (60.1 ± 11.3 years) was included for testing. A DLCR model was proposed to predict the expression of Ki-67 with cMRI, including T2W, DW, and dynamic contrast enhancement (DCE) images as inputs. The images of the shear wave speed (c-map) and phase angle (φ-map) derived from MRE were also fed into the DLCR model. The Ki-67 expression was classified into low and high groups with a threshold of 20%. Both c and φ values were ranked within the top six features for Ki-67 prediction with random forest selection, which revealed the value of MRE-based viscosity for the assessment of tumor proliferation status in HCC. When comparing the six CNN models, Xception showed the best performance for classifying the Ki-67 expression, with an AUC of 0.80 ± 0.03 (CI: 0.79-0.81) and accuracy of 0.77 ± 0.04 (CI: 0.76-0.78) when cMRI were fed into the model. The model with all modalities (MRE, AFP, and cMRI) as inputs achieved the highest AUC of 0.90 ± 0.03 (CI: 0.89-0.91) in the validation cohort. The same finding was observed in the independent testing cohort, with an AUC of 0.83 ± 0.03 (CI: 0.82-0.84). The shear wave speed and phase angle improved the performance of the DLCR model significantly for Ki-67 prediction, suggesting that MRE-based c and φ-maps can serve as important parameters to assess the tumor proliferation status in HCC.

MAGnitude-Image-to-Complex K-space (MAGIC-K) Net: A Data Augmentation Network for Image Reconstruction

Deep learning has demonstrated superior performance in image reconstruction compared to most conventional iterative algorithms. However, their effectiveness and generalization capability are highly dependent on the sample size and diversity of the training data. Deep learning-based reconstruction requires multi-coil raw k-space data, which are not collected by routine scans. On the other hand, large amounts of magnitude images are readily available in hospitals. Hence, we proposed the MAGnitude Images to Complex K-space (MAGIC-K) Net to generate multi-coil k-space data from existing magnitude images and a limited number of required raw k-space data to facilitate the reconstruction. Compared to some basic data augmentation methods applying global intensity and displacement transformations to the source images, the MAGIC-K Net can generate more realistic intensity variations and displacements from pairs of anatomical Digital Imaging and Communications in Medicine (DICOM) images. The reconstruction performance was validated in 30 healthy volunteers and 6 patients with different types of tumors. The experimental results demonstrated that the high-resolution Diffusion Weighted Image (DWI) reconstruction benefited from the proposed augmentation method. The MAGIC-K Net enabled the deep learning network to reconstruct images with superior performance in both healthy and tumor patients, qualitatively and quantitatively.

Current status of deep learning in abdominal image reconstruction

Abdominal magnetic resonance imaging (MRI) and computed tomography (CT) are commonly used for disease screening, diagnosis, and treatment guidance. However, abdominal MRI has disadvantages including slow speed and vulnerability to motions, while CT suffers from problems of radiation. It has been reported that deep learning reconstruction can solve such problems while maintaining good image quality. Recently, deep learning-based image reconstruction has become a hot topic in the field of medical imaging. This study reviews the latest research on deep learning reconstruction in abdominal imaging, including the widely used convolutional neural network, generative adversarial network, and recurrent neural network.

Recommendation for Cardiac Magnetic Resonance Imaging-Based Phenotypic Study: Imaging Part

Cardiac magnetic resonance (CMR) imaging provides important biomarkers for the early diagnosis of many cardiovascular diseases and has been reported to reveal phenome-wide associations of cardiac/aortic structure and functionality in population studies. Nevertheless, due to the complexity of operation and variations among manufactural vendors, magnetic field strengths, coils, sequences, scan parameters, and image analysis approaches, CMR is rarely used in large cohort studies. Existing guidelines mainly focused on the diagnosis of cardiovascular diseases, which did not aim to basic research. The purpose of this study was to propose a recommendation for CMR based phenotype measurements for cohort study. We classify the imaging sequences of CMR into three categories according to the importance and universality of corresponding measurable phenotypes. The acquisition time and repeatability of the phenotypic measurement were also taken into consideration during the categorization. Unlike other guidelines, this recommendation focused on quantitative measurement of large amount of phenotypes from CMR.

Transfer learning enhanced generative adversarial networks for multi-channel MRI reconstruction

Deep learning based generative adversarial networks (GAN) can effectively perform image reconstruction with under-sampled MR data. In general, a large number of training samples are required to improve the reconstruction performance of a certain model. However, in real clinical applications, it is difficult to obtain tens of thousands of raw patient data to train the model since saving k-space data is not in the routine clinical flow. Therefore, enhancing the generalizability of a network based on small samples is urgently needed. In this study, three novel applications were explored based on parallel imaging combined with the GAN model (PI-GAN) and transfer learning. The model was pre-trained with public Calgary brain images and then fine-tuned for use in (1) patients with tumors in our center; (2) different anatomies, including knee and liver; (3) different k-space sampling masks with acceleration factors (AFs) of 2 and 6. As for the brain tumor dataset, the transfer learning results could remove the artifacts found in PI-GAN and yield smoother brain edges. The transfer learning results for the knee and liver were superior to those of the PI-GAN model trained with its own dataset using a smaller number of training cases. However, the learning procedure converged more slowly in the knee datasets compared to the learning in the brain tumor datasets. The reconstruction performance was improved by transfer learning both in the models with AFs of 2 and 6. Of these two models, the one with AF = 2 showed better results. The results also showed that transfer learning with the pre-trained model could solve the problem of inconsistency between the training and test datasets and facilitate generalization to unseen data.

Imaging-Based Staging of Hepatic Fibrosis in Patients with Hepatitis B: A Dynamic Radiomics Model Based on Gd-EOB-DTPA-Enhanced MRI

Accurate grading of liver fibrosis can effectively assess the severity of liver disease and help doctors make an appropriate diagnosis. This study aimed to perform the automatic staging of hepatic fibrosis on patients with hepatitis B, who underwent gadolinium ethoxybenzyl diethylenetriamine pentaacetic acid (Gd-EOB-DTPA)-enhanced magnetic resonance imaging with dynamic radiomics analysis. The proposed dynamic radiomics model combined imaging features from multi-phase dynamic contrast-enhanced (DCE) images and time-domain information. Imaging features were extracted from the deep learning-based segmented liver volume, and time-domain features were further explored to analyze the variation in features during contrast enhancement. Model construction and evaluation were based on a 132-case data set. The proposed model achieved remarkable performance in significant fibrosis (fibrosis stage S1 vs. S2–S4; accuracy (ACC) = 0.875, area under the curve (AUC) = 0.867), advanced fibrosis (S1–S2 vs. S3–S4; ACC = 0.825, AUC = 0.874), and cirrhosis (S1–S3 vs. S4; ACC = 0.850, AUC = 0.900) classifications in the test set. It was more dominant compared with the conventional single-phase or multi-phase DCE-based radiomics models, normalized liver enhancement, and some serological indicators. Time-domain features were found to play an important role in the classification models. The dynamic radiomics model can be applied for highly accurate automatic hepatic fibrosis staging.

PIC-GAN: A Parallel Imaging Coupled Generative Adversarial Network for Accelerated Multi-Channel MRI Reconstruction

In this study, we proposed a model combing parallel imaging (PI) with generative adversarial network (GAN) architecture (PIC-GAN) for accelerated multi-channel magnetic resonance imaging (MRI) reconstruction. This model integrated data fidelity and regularization terms into the generator to benefit from multi-coils information and provide an "end-to-end" reconstruction. Besides, to better preserve image details during reconstruction, we combined the adversarial loss with pixel-wise loss in both image and frequency domains. The proposed PIC-GAN framework was evaluated on abdominal and knee MRI images using 2, 4 and 6-fold accelerations with different undersampling patterns. The performance of the PIC-GAN was compared to the sparsity-based parallel imaging (L1-ESPIRiT), the variational network (VN), and conventional GAN with single-channel images as input (zero-filled (ZF)-GAN). Experimental results show that our PIC-GAN can effectively reconstruct multi-channel MR images at a low noise level and improved structure similarity of the reconstructed images. PIC-GAN has yielded the lowest Normalized Mean Square Error (in ×10-5) (PIC-GAN: 0.58 ± 0.37, ZF-GAN: 1.93 ± 1.41, VN: 1.87 ± 1.28, L1-ESPIRiT: 2.49 ± 1.04 for abdominal MRI data and PIC-GAN: 0.80 ± 0.26, ZF-GAN: 0.93 ± 0.29, VN:1.18 ± 0.31, L1-ESPIRiT: 1.28 ± 0.24 for knee MRI data) and the highest Peak Signal to Noise Ratio (PIC-GAN: 34.43 ± 1.92, ZF-GAN: 31.45 ± 4.0, VN: 29.26 ± 2.98, L1-ESPIRiT: 25.40 ± 1.88 for abdominal MRI data and PIC-GAN: 34.10 ± 1.09, ZF-GAN: 31.47 ± 1.05, VN: 30.01 ± 1.01, L1-ESPIRiT: 28.01 ± 0.98 for knee MRI data) compared to ZF-GAN, VN and L1-ESPIRiT with an under-sampling factor of 6. The proposed PIC-GAN framework has shown superior reconstruction performance in terms of reducing aliasing artifacts and restoring tissue structures as compared to other conventional and state-of-the-art reconstruction methods.

Parallel imaging with a combination of sensitivity encoding and generative adversarial networks

Background

Magnetic resonance imaging (MRI) has the limitation of low imaging speed. Acceleration methods using under-sampled k-space data have been widely exploited to improve data acquisition without reducing the image quality. Sensitivity encoding (SENSE) is the most commonly used method for multi-channel imaging. However, SENSE has the drawback of severe g-factor artifacts when the under-sampling factor is high. This paper applies generative adversarial networks (GAN) to remove g-factor artifacts from SENSE reconstructions.

Methods

Our method was evaluated on a public knee database containing 20 healthy participants. We compared our method with conventional GAN using zero-filled (ZF) images as input. Structural similarity (SSIM), peak signal to noise ratio (PSNR), and normalized mean square error (NMSE) were calculated for the assessment of image quality. A paired student's t-test was conducted to compare the image quality metrics between the different methods. Statistical significance was considered at P<0.01.

Results

The proposed method outperformed SENSE, variational network (VN), and ZF + GAN methods in terms of SSIM (SENSE + GAN: 0.81±0.06, SENSE: 0.40±0.07, VN: 0.79±0.06, ZF + GAN: 0.77±0.06), PSNR (SENSE + GAN: 31.90±1.66, SENSE: 22.70±1.99, VN: 31.35±2.01, ZF + GAN: 29.95±1.59), and NMSE (×10^-7) (SENSE + GAN: 0.95±0.34, SENSE: 4.81±1.33, VN: 0.97±0.30, ZF + GAN: 1.60±0.84) with an under-sampling factor of up to 6-fold.

Conclusions

This study demonstrated the feasibility of using GAN to improve the performance of SENSE reconstruction. The improvement of reconstruction is more obvious for higher under-sampling rates, which shows great potential for many clinical applications.