Generation of abdominal synthetic CTs from 0.35T MR images using generative adversarial networks for MR-only liver radiotherapy

A systematic literature review: deep learning techniques for synthetic medical image generation and their applications in radiotherapy

The aim of this systematic review is to determine whether Deep Learning (DL) algorithms can provide a clinically feasible alternative to classic algorithms for synthetic Computer Tomography (sCT). The following categories are presented in this study: ∙ MR-based treatment planning and synthetic CT generation techniques. ∙ Generation of synthetic CT images based on Cone Beam CT images. ∙ Low-dose CT to High-dose CT generation. ∙ Attenuation correction for PET images. To perform appropriate database searches, we reviewed journal articles published between January 2018 and June 2023. Current methodology, study strategies, and results with relevant clinical applications were analyzed as we outlined the state-of-the-art of deep learning based approaches to inter-modality and intra-modality image synthesis. This was accomplished by contrasting the provided methodologies with traditional research approaches. The key contributions of each category were highlighted, specific challenges were identified, and accomplishments were summarized. As a final step, the statistics of all the cited works from various aspects were analyzed, which revealed that DL-based sCTs have achieved considerable popularity, while also showing the potential of this technology. In order to assess the clinical readiness of the presented methods, we examined the current status of DL-based sCT generation.

Deep learning based synthesis of MRI, CT and PET: Review and analysis

Medical image synthesis represents a critical area of research in clinical decision-making, aiming to overcome the challenges associated with acquiring multiple image modalities for an accurate clinical workflow. This approach proves beneficial in estimating an image of a desired modality from a given source modality among the most common medical imaging contrasts, such as Computed Tomography (CT), Magnetic Resonance Imaging (MRI), and Positron Emission Tomography (PET). However, translating between two image modalities presents difficulties due to the complex and non-linear domain mappings. Deep learning-based generative modelling has exhibited superior performance in synthetic image contrast applications compared to conventional image synthesis methods. This survey comprehensively reviews deep learning-based medical imaging translation from 2018 to 2023 on pseudo-CT, synthetic MR, and synthetic PET. We provide an overview of synthetic contrasts in medical imaging and the most frequently employed deep learning networks for medical image synthesis. Additionally, we conduct a detailed analysis of each synthesis method, focusing on their diverse model designs based on input domains and network architectures. We also analyse novel network architectures, ranging from conventional CNNs to the recent Transformer and Diffusion models. This analysis includes comparing loss functions, available datasets and anatomical regions, and image quality assessments and performance in other downstream tasks. Finally, we discuss the challenges and identify solutions within the literature, suggesting possible future directions. We hope that the insights offered in this survey paper will serve as a valuable roadmap for researchers in the field of medical image synthesis.

Machine Learning for Medical Image Translation: A Systematic Review

BACKGROUND

CT scans are often the first and only form of brain imaging that is performed to inform treatment plans for neurological patients due to its time- and cost-effective nature. However, MR images give a more detailed picture of tissue structure and characteristics and are more likely to pick up abnormalities and lesions. The purpose of this paper is to review studies which use deep learning methods to generate synthetic medical images of modalities such as MRI and CT.

METHODS

A literature search was performed in March 2023, and relevant articles were selected and analyzed. The year of publication, dataset size, input modality, synthesized modality, deep learning architecture, motivations, and evaluation methods were analyzed.

RESULTS

A total of 103 studies were included in this review, all of which were published since 2017. Of these, 74% of studies investigated MRI to CT synthesis, and the remaining studies investigated CT to MRI, Cross MRI, PET to CT, and MRI to PET. Additionally, 58% of studies were motivated by synthesizing CT scans from MRI to perform MRI-only radiation therapy. Other motivations included synthesizing scans to aid diagnosis and completing datasets by synthesizing missing scans.

CONCLUSIONS

Considerably more research has been carried out on MRI to CT synthesis, despite CT to MRI synthesis yielding specific benefits. A limitation on medical image synthesis is that medical datasets, especially paired datasets of different modalities, are lacking in size and availability; it is therefore recommended that a global consortium be developed to obtain and make available more datasets for use. Finally, it is recommended that work be carried out to establish all uses of the synthesis of medical scans in clinical practice and discover which evaluation methods are suitable for assessing the synthesized images for these needs.

Patient-specific quality assurance strategies for synthetic computed tomography in magnetic resonance-only radiotherapy of the abdomen

Background and purpose

The superior tissue contrast of magnetic resonance (MR) compared to computed tomography (CT) led to an increasing interest towards MR-only radiotherapy. For the latter, the dose calculation should be performed on a synthetic CT (sCT). Patient-specific quality assurance (PSQA) methods have not been established yet and this study aimed to assess several software-based solutions.

Materials and methods

A retrospective study was performed on 20 patients treated at an MR-Linac, which were selected to evenly cover four subcategories: (i) standard, (ii) air pockets, (iii) lung and (iv) implant cases. The neural network (NN) CycleGAN was adopted to generate a reference sCT, which was then compared to four PSQA methods: (A) water override of body, (B) five tissue classes with bulk densities, (C) sCT generated by a separate NN (pix2pix) and (D) deformed CT.

Results

The evaluation of the dose endpoints demonstrated that while all methods A-D provided statistically equivalent results (p = 0.05) within the 2% level for the standard cases (i), only the methods C-D guaranteed the same result over the whole cohort. The bulk densities override was shown to be a valuable method in absence of lung tissue within the beam path.

Conclusion

The observations of this study suggested that the use of an additional sCT generated by a separate NN was an appropriate tool to perform PSQA of a sCT in an MR-only workflow at an MR-Linac. The time and dose endpoints requirements were respected, namely within 10 min and 2%.

Synthetic computed tomography for low-field magnetic resonance-only radiotherapy in head-and-neck cancer using residual vision transformers

Background and purpose

Synthetic computed tomography (sCT) scans are necessary for dose calculation in magnetic resonance (MR)-only radiotherapy. While deep learning (DL) has shown remarkable performance in generating sCT scans from MR images, research has predominantly focused on high-field MR images. This study presents the first implementation of a DL model for sCT generation in head-and-neck (HN) cancer using low-field MR images. Specifically, the use of vision transformers (ViTs) was explored.

Materials and methods

The dataset consisted of 31 patients, resulting in 196 pairs of deformably-registered computed tomography (dCT) and MR scans. The latter were obtained using a balanced steady-state precession sequence on a 0.35T scanner. Residual ViTs were trained on 2D axial, sagittal, and coronal slices, respectively, and the final sCTs were generated by averaging the models' outputs. Different image similarity metrics, dose volume histogram (DVH) deviations, and gamma analyses were computed on the test set (n = 6). The overlap between auto-contours on sCT scans and manual contours on MR images was evaluated for different organs-at-risk using the Dice score.

Results

The median [range] value of the test mean absolute error was 57 [37-74] HU. DVH deviations were below 1% for all structures. The median gamma passing rates exceeded 94% in the 2%/2mm analysis (threshold = 90%). The median Dice scores were above 0.7 for all organs-at-risk.

Conclusions

The clinical applicability of DL-based sCT generation from low-field MR images in HN cancer was proved. High sCT-dCT similarity and dose metric accuracy were achieved, and sCT suitability for organs-at-risk auto-delineation was shown.

Evaluation of MRI-only based online adaptive radiotherapy of abdominal region on MR-linac

PURPOSE

A hybrid magnetic resonance linear accelerator (MRL) can perform magnetic resonance imaging (MRI) with high soft-tissue contrast to be used for online adaptive radiotherapy (oART). To obtain electron densities needed for the oART dose calculation, a computed tomography (CT) is often deformably registered to MRI. Our aim was to evaluate an MRI-only based synthetic CT (sCT) generation as an alternative to the deformed CT (dCT)-based oART in the abdominal region.

METHODS

The study data consisted of 57 patients who were treated on a 0.35 T MRL system mainly for abdominal tumors. Simulation MRI-CT pairs of 43 patients were used for training and validation of a prototype convolutional neural network sCT-generation algorithm, based on HighRes3DNet, for the abdominal region. For remaining test patients, sCT images were produced from simulation MRIs and daily MRIs. The dCT-based plans were re-calculated on sCT with identical calculation parameters. The sCT and dCT were compared in terms of geometric agreement and calculated dose.

RESULTS

The mean and one standard deviation of the geometric agreement metrics over dCT-sCT-pairs were: mean error of 8 ± 10 HU, mean absolute error of 49 ± 10 HU, and Dice similarity coefficient of 55 ± 12%, 60 ± 5%, and 82 ± 15% for bone, fat, and lung tissues, respectively. The dose differences between the sCT and dCT-based dose for planning target volumes were 0.5 ± 0.9%, 0.6 ± 0.8%, and 0.5 ± 0.8% at D_2% , D_50% , and D_98% in physical dose and 0.8 ± 1.4%, 0.8 ± 1.2%, and 0.6 ± 1.1% in biologically effective dose (BED). For organs-at-risk, the dose differences of all evaluated dose-volume histogram points were within [-4.5%, 7.8%] and [-1.1 Gy, 3.5 Gy] in both physical dose and BED.

CONCLUSIONS

The geometric agreement metrics were within typically reported values and most average relative dose differences were within 1%. Thus, an MRI-only sCT-based approach is a promising alternative to the current clinical practice of the abdominal oART on MRL.

Synthetic CT in Carbon Ion Radiotherapy of the Abdominal Site

The generation of synthetic CT for carbon ion radiotherapy (CIRT) applications is challenging, since high accuracy is required in treatment planning and delivery, especially in an anatomical site as complex as the abdomen. Thirty-nine abdominal MRI-CT volume pairs were collected and a three-channel cGAN (accounting for air, bones, soft tissues) was used to generate sCTs. The network was tested on five held-out MRI volumes for two scenarios: (i) a CT-based segmentation of the MRI channels, to assess the quality of sCTs and (ii) an MRI manual segmentation, to simulate an MRI-only treatment scenario. The sCTs were evaluated by means of similarity metrics (e.g., mean absolute error, MAE) and geometrical criteria (e.g., dice coefficient). Recalculated CIRT plans were evaluated through dose volume histogram, gamma analysis and range shift analysis. The CT-based test set presented optimal MAE on bones (86.03 ± 10.76 HU), soft tissues (55.39 ± 3.41 HU) and air (54.42 ± 11.48 HU). Higher values were obtained from the MRI-only test set (MAE_BONE = 154.87 ± 22.90 HU). The global gamma pass rate reached 94.88 ± 4.9% with 3%/3 mm, while the range shift reached a median (IQR) of 0.98 (3.64) mm. The three-channel cGAN can generate acceptable abdominal sCTs and allow for CIRT dose recalculations comparable to the clinical plans.

A body mass index-based method for "MR-only" abdominal MR-guided adaptive radiotherapy

PURPOSE

Dose calculation for MR-guided radiotherapy (MRgRT) at the 0.35 T MR-Linac is currently based on deformation of planning CTs (defCT) acquired for each patient. We present a simple and robust bulk density overwrite synthetic CT (sCT) method for abdominal treatments in order to streamline clinical workflows.

METHOD

Fifty-six abdominal patient treatment plans were retrospectively evaluated. All patients had been treated at the MR-Linac using MR datasets for treatment planning and plan adaption and defCT for dose calculation. Bulk density CTs (4M-sCT) were generated from MR images with four material compartments (bone, lung, air, soft tissue). The relative electron densities (RED) for bone and lung were extracted from contoured CT structure average REDs. For soft tissue, a correlation between BMI and RED was evaluated. Dose was recalculated on 4M-sCT and compared to dose distributions on defCTs assessing dose differences in the PTV and organs at risk (OAR).

RESULTS

Mean RED of bone was 1.17 ± 0.02, mean RED of lung 0.17 ± 0.05. The correlation between BMI and RED for soft tissue was statistically significant (p < 0.01). PTV dose differences between 4M-sCT and defCT were Dmean: -0.4 ± 1.0%, D1%: -0.3 ± 1.1% and D95%: -0.5 ± 1.0%. OARs showed D2%: -0.3 ± 1.9% and Dmean: -0.1 ± 1.4% differences. Local 3D gamma index pass rates (2%/2mm) between dose calculated using 4M-sCT and defCT were 96.8 ± 2.6% (range 89.9-99.6%).

CONCLUSION

The presented method for sCT generation enables precise dose calculation for MR-only abdominal MRgRT.

QACL: Quartet attention aware closed-loop learning for abdominal MR-to-CT synthesis via simultaneous registration

Synthesis of computed tomography (CT) images from magnetic resonance (MR) images is an important task to overcome the lack of electron density information in MR-only radiotherapy treatment planning (RTP). Some innovative methods have been proposed for abdominal MR-to-CT synthesis. However, it is still challenging due to the large misalignment between preprocessed abdominal MR and CT images and the insufficient feature information learned by models. Although several studies have used the MR-to-CT synthesis to alleviate the difficulty of multi-modal registration, this misalignment remains unsolved when training the MR-to-CT synthesis model. In this paper, we propose an end-to-end quartet attention aware closed-loop learning (QACL) framework for MR-to-CT synthesis via simultaneous registration. Specifically, the proposed quartet attention generator and mono-modal registration network form a closed-loop to improve the performance of MR-to-CT synthesis via simultaneous registration. In particular, a quartet-attention mechanism is developed to enlarge the receptive fields in networks to extract the long-range and cross-dimension spatial dependencies. Experimental results on two independent abdominal datasets demonstrate that our QACL achieves impressive results with MAE of 55.30±10.59 HU, PSNR of 22.85±1.43 dB, and SSIM of 0.83±0.04 for synthesis, and with Dice of 0.799±0.129 for registration. The proposed QACL outperforms the state-of-the-art MR-to-CT synthesis and multi-modal registration methods.

Synthetic computed tomography generation for abdominal adaptive radiotherapy using low-field magnetic resonance imaging

Background and Purpose

Magnetic Resonance guided Radiotherapy (MRgRT) still needs the acquisition of Computed Tomography (CT) images and co-registration between CT and Magnetic Resonance Imaging (MRI). The generation of synthetic CT (sCT) images from the MR data can overcome this limitation. In this study we aim to propose a Deep Learning (DL) based approach for sCT image generation for abdominal Radiotherapy using low field MR images.

Materials and methods

CT and MR images were collected from 76 patients treated on abdominal sites. U-Net and conditional Generative Adversarial Network (cGAN) architectures were used to generate sCT images. Additionally, sCT images composed of only six bulk densities were generated with the aim of having a Simplified sCT.Radiotherapy plans calculated using the generated images were compared to the original plan in terms of gamma pass rate and Dose Volume Histogram (DVH) parameters.

Results

sCT images were generated in 2 s and 2.5 s with U-Net and cGAN architectures respectively.Gamma pass rates for 2%/2mm and 3%/3mm criteria were 91% and 95% respectively. Dose differences within 1% for DVH parameters on the target volume and organs at risk were obtained.

Conclusion

U-Net and cGAN architectures are able to generate abdominal sCT images fast and accurately from low field MRI.

Investigation of a water equivalent depth method for dosimetric accuracy evaluation of synthetic CT

PURPOSE

To provide a metric that reflects the dosimetric utility of the synthetic CT (sCT) and can be rapidly determined.

METHODS

Retrospective CT and atlas-based sCT of 62 (53 IMRT and 9 VMAT) prostate cancer patients were used. For image similarity measurements, the sCT and reference CT (rCT) were aligned using clinical registration parameters. Conventional image similarity metrics including the mean absolute error (MAE) and mean error (ME) were calculated. The water equivalent depth (WED) was automatically determined for each patient on the rCT and sCT as the distance from the skin surface to the treatment plan isocentre at 36 equidistant gantry angles, and the mean WED difference (ΔWED¯) between the two scans was calculated. Doses were calculated on each scan pair for the clinical plan in the treatment planning system. The image similarity measurements and ΔWED¯ were then compared to the isocentre dose difference (ΔD_iso) between the two scans.

RESULTS

While no particular relationship to dose was observed for the other image similarity metrics, the ME results showed a linear trend against ΔD_iso with R² = 0.6, and the 95 % prediction interval for ΔD_iso between -1.2 and 1 %. The ΔWED¯ results showed an improved linear trend (R² = 0.8) with a narrower 95 % prediction interval from -0.8 % to 0.8 %.

CONCLUSION

ΔWED¯ highly correlates with ΔD_iso for the reference and synthetic CT scans. This is easy to calculate automatically and does not require time-consuming dose calculations. Therefore, it can facilitate the process of developing and evaluating new sCT generation algorithms.

Towards 2D/3D Registration of the Preoperative MRI to Intraoperative Fluoroscopic Images for Visualization of Bone Defects

Magnetic Resonance Imaging (MRI) is a medical imaging modality that allows for the evaluation of soft-tissue diseases and the assessment of bone quality. Preoperative MRI volumes are used by surgeons to identify defected bones, perform the segmentation of lesions, and generate surgical plans before the surgery. Nevertheless, conventional intraoperative imaging modalities such as fluoroscopy are less sensitive in detecting potential lesions. In this work, we propose a 2D/3D registration pipeline that aims to register preoperative MRI with intraoperative 2D fluoroscopic images. To showcase the feasibility of our approach, we use the core decompression procedure as a surgical example to perform 2D/3D femur registration. The proposed registration pipeline is evaluated using digitally reconstructed radiographs (DRRs) to simulate the intraoperative fluoroscopic images. The resulting transformation from the registration is later used to create overlays of preoperative MRI annotations and planning data to provide intraoperative visual guidance to surgeons. Our results suggest that the proposed registration pipeline is capable of achieving reasonable transformation between MRI and digitally reconstructed fluoroscopic images for intraoperative visualization applications.

Image Translation by Ad CycleGAN for COVID-19 X-Ray Images: A New Approach for Controllable GAN

We propose a new generative model named adaptive cycle-consistent generative adversarial network, or Ad CycleGAN to perform image translation between normal and COVID-19 positive chest X-ray images. An independent pre-trained criterion is added to the conventional Cycle GAN architecture to exert adaptive control on image translation. The performance of Ad CycleGAN is compared with the Cycle GAN without the external criterion. The quality of the synthetic images is evaluated by quantitative metrics including Mean Squared Error (MSE), Root Mean Squared Error (RMSE), Peak Signal-to-Noise Ratio (PSNR), Universal Image Quality Index (UIQI), visual information fidelity (VIF), Frechet Inception Distance (FID), and translation accuracy. The experimental results indicate that the synthetic images generated either by the Cycle GAN or by the Ad CycleGAN have lower MSE and RMSE, and higher scores in PSNR, UIQI, and VIF in homogenous image translation (i.e., Y → Y) compared to the heterogenous image translation process (i.e., X → Y). The synthetic images by Ad CycleGAN through the heterogeneous image translation have significantly higher FID score compared to Cycle GAN (p < 0.01). The image translation accuracy of Ad CycleGAN is higher than that of Cycle GAN when normal images are converted to COVID-19 positive images (p < 0.01). Therefore, we conclude that the Ad CycleGAN with the independent criterion can improve the accuracy of GAN image translation. The new architecture has more control on image synthesis and can help address the common class imbalance issue in machine learning methods and artificial intelligence applications with medical images.

A deep learning approach to generate synthetic CT in low field MR-guided radiotherapy for lung cases

INTRODUCTION

This study aims to apply a conditional Generative Adversarial Network (cGAN) to generate synthetic Computed Tomography (sCT) from 0.35 Tesla Magnetic Resonance (MR) images of the thorax.

METHODS

Sixty patients treated for lung lesions were enrolled and divided into training (32), validation (8), internal (10,T_A) and external (10,T_B) test set. Image accuracy of generated sCT was evaluated computing the mean absolute (MAE) and mean error (ME) with respect the original CT. Three treatment plans were calculated for each patient considering MRI as reference image: original CT, sCT (pure sCT) and sCT with GTV density override (hybrid sCT) were used as Electron Density (ED) map. Dose accuracy was evaluated comparing treatment plans in terms of gamma analysis and Dose Volume Histogram (DVH) parameters.

RESULTS

No significant difference was observed between the test sets for image and dose accuracy parameters. Considering the whole test cohort, a MAE of 54.9 ± 10.5 HU and a ME of 4.4 ± 7.4 HU was obtained. Mean gamma passing rates for 2%/2mm, and 3%/3mm tolerance criteria were 95.5 ± 5.9% and 98.2 ± 4.1% for pure sCT, 96.1 ± 5.1% and 98.5 ± 3.9% for hybrid sCT: the difference between the two approaches was significant (p = 0.01). As regards DVH analysis, differences in target parameters estimation were found to be within 5% using hybrid approach and 20% using pure sCT.

CONCLUSION

The DL algorithm here presented can generate sCT images in the thorax with good image and dose accuracy, especially when the hybrid approach is used. The algorithm does not suffer from inter-scanner variability, making feasible the implementation of MR-only workflows for palliative treatments.

Personalized modeling to improve pseudo-CT images for magnetic resonance imaging-guided adaptive radiotherapy

PURPOSE

Magnetic resonance imaging-guided adaptive radiotherapy (MRIgART) greatly improves daily tumor localization and enables online re-planning to obtain maximum dosimetric benefits. However, accurately predicting patient-specific electron density maps for adaptive radiotherapy (ART) planning remains a challenge. Therefore, this study proposes a personalized modeling framework for generating pseudo-computed tomography (pCT) in MRIgART.

METHODS AND MATERIALS

Eighty-three patients who received MRIgART were included and CT simulations were performed on all the patients. Daily T2-weighted 1.5 T MRI was acquired using the Unity MR-linac for adaptive planning. Pairs of co-registered CT and daily MRI images of the randomly selected training set (68 patients) were inputted into a generative adversarial network (GAN) to establish a population model. The personalized model for each patient in the test set (15 patients) was acquired using model fine-tuning, which adopted the pair of the deformable-registered CT and the first daily MRI to fine-tune the population model. The pCT quality was quantitatively evaluated in the second and the last fractions with three metrics: intensity accuracy using mean absolute error (MAE); anatomical structure similarity using dice similarity coefficient (DSC); and dosimetric consistency using gamma-passing rate (GPR).

RESULTS

The image generation speed was 65 slices per second. For the last fractions, and for head-neck, thoracoabdominal, and pelvic cases, the average MAEs were 76.8 HU vs. 123.6 HU, 38.1 HU vs. 52.0 HU, and 29.5 HU vs. 39.7 HU, respectively. Furthermore, the average DSCs of bone were 0.92 vs. 0.80, 0.85 vs. 0.73, and 0.94 vs. 0.88; and the average GPRs (1%/1 mm) were 95.5% vs. 84.7%, 97.7% vs. 92.8%, and 95.5% vs. 88.7%, for personalized vs. population models, respectively. Results of the second fractions were similar.

CONCLUSIONS

The proposed personalized modeling framework remarkably improved pCT quality for multiple treatment sites and was well suited for the MRIgART clinical setting.

Multi parametric magnetic resonance imaging for radiation treatment planning

In recent years, multi-parametric magnetic resonance imaging (MpMRI) has played a major role in radiation therapy treatment planning. The superior soft tissue contrast, functional or physiological imaging capabilities and the flexibility of site-specific image sequence development has placed MpMRI at the forefront. In this article, the present status of MpMRI for external beam radiation therapy planning is reviewed. Common MpMRI sequences, preprocessing and QA strategies are briefly discussed, and various image registration techniques and strategies are addressed. Image segmentation methods including automatic segmentation and deep learning techniques for organs at risk and target delineation are reviewed. Due to the advancement in MRI guided online adaptive radiotherapy, treatment planning considerations addressing MRI only planning are also discussed. This article is protected by copyright. All rights reserved.

Synthetic Computed Tomography Generation from 0.35T Magnetic Resonance Images for Magnetic Resonance-Only Radiation Therapy Planning Using Perceptual Loss Models

PURPOSE

Magnetic resonance imaging (MRI) provides excellent soft-tissue contrast, which makes it useful for delineating tumor and normal structures in radiation therapy planning, but MRI cannot readily provide electron density for dose calculation. Computed tomography (CT) is used but introduces registration uncertainty between MRI and CT. Previous studies have shown that synthetic CTs (sCTs) can be generated directly from MRI images with deep learning methods. However, mainly high-field MRI images have been validated. This study tested whether acceptable sCTs for MR-only radiation therapy planning can be synthesized using an integrated MR-guided linear accelerator at 0.35T, using MRI images and treatment plans in the liver region.

METHODS AND MATERIALS

Two models were investigated in this study: a convolutional neural network (Unet) with conventional mean square error (MSE) loss and a Unet using a secondary convolutional neural network for perceptual loss. A total of 37 cases were used in this study with 10-fold cross validation, and 37 treatment plans were generated and evaluated for target coverage and dose to organs at risk (OARs) in the MSE loss model, perceptual loss model, and original CT.

RESULTS

The sCTs predicted by the perceptual loss model had improved subjective visual quality compared with those predicted by the MSE loss model, but both were similar in mean absolute error (MAE), peak-signal-to-noise ratio (PSNR), and normalized cross-correlation (NCC). The MAE, PSNR, and NCC for the perceptual loss model were 35.64, 24.11, and 0.9539, respectively, and those for the MSE loss model were 35.67, 24.36, and 0.9566, respectively. No significant differences in target coverage and dose to OARs were found between the sCT predicted by the perceptual loss model or by the MSE model and the original CT image.

CONCLUSIONS

This study indicated that a Unet with both MSE loss and perceptual loss models can be used for generating sCT images from a 0.35T integrated MR linear accelerator.

An MRI sequence independent convolutional neural network for synthetic head CT generation in proton therapy

A magnetic resonance imaging (MRI) sequence independent deep learning technique was developed and validated to generate synthetic computed tomography (sCT) scans for MR guided proton therapy. 47 meningioma patients previously undergoing proton therapy based on pencil beam scanning were divided into training (33), validation (6), and test (8) cohorts. T₁, T₂, and contrast enhanced T₁ (T1CM) MRI sequences were used in combination with the planning CT (pCT) data to train a 3D U-Net architecture with ResNet-Blocks. A hyperparameter search was performed including two loss functions, two group sizes of normalisation, and depth of the network. Training outcome was compared between models trained for each individual MRI sequence and for all sequences combined. The performance was evaluated based on a metric and dosimetric analysis as well as spot difference maps. Furthermore, the influence of immobilisation masks that are not visible on MRIs was investigated. Based on the hyperparameter search, the final model was trained with fixed features per group for the group normalisation, six down-convolution steps, an input size of 128×192×192, and feature loss. For the test dataset for body/bone the mean absolute error (MAE) values were on average 79.8/216.3Houndsfield unit (HU) when trained using T1 images, 71.1/186.1HU for T2, and 82.9/236.4HU for T1CM. The structural similarity metric (SSIM) ranged from 0.95 to 0.98 for all sequences. The investigated dose parameters of the target structures agreed within 1% between original proton treatment plans and plans recalculated on sCTs. The spot difference maps had peaks at ±0.2cm and for 98% of all spots the difference was less than 1cm. A novel MRI sequence independent sCT generator was developed, which suggests that the training phase of neural networks can be disengaged from specific MRI acquisition protocols. In contrast to previous studies, the patient cohort consisted exclusively of actual proton therapy patients (i.e. "real-world data").

Abdominal synthetic CT reconstruction with intensity projection prior for MRI-only adaptive radiotherapy

Objective. Owing to the superior soft tissue contrast of MRI, MRI-guided adaptive radiotherapy (ART) is well-suited to managing interfractional changes in anatomy. An MRI-only workflow is desirable, but producing synthetic CT (sCT) data through paired data-driven deep learning (DL) for abdominal dose calculations remains a challenge due to the highly variable presence of intestinal gas. We present the preliminary dosimetric evaluation of our novel approach to sCT reconstruction that is well suited to handling intestinal gas in abdominal MRI-only ART.Approach. We utilize a paired data DL approach enabled by the intensity projection prior, in which well-matching training pairs are created by propagating air from MRI to corresponding CT scans. Evaluations focus on two classes: patients with (1) little involvement of intestinal gas, and (2) notable differences in intestinal gas presence between corresponding scans. Comparisons between sCT-based plans and CT-based clinical plans for both classes are made at the first treatment fraction to highlight the dosimetric impact of the variable presence of intestinal gas.Main results. Class 1 patients (n= 13) demonstrate differences in prescribed dose coverage of the PTV of 1.3 ± 2.1% between clinical plans and sCT-based plans. Mean DVH differences in all structures for Class 1 patients are found to be statistically insignificant. In Class 2 (n= 20), target coverage is 13.3 ± 11.0% higher in the clinical plans and mean DVH differences are found to be statistically significant.Significance. Significant deviations in calculated doses arising from the variable presence of intestinal gas in corresponding CT and MRI scans result in uncertainty in high-dose regions that may limit the effectiveness of adaptive dose escalation efforts. We have proposed a paired data-driven DL approach to sCT reconstruction for accurate dose calculations in abdominal ART enabled by the creation of a clinically unavailable training data set with well-matching representations of intestinal gas.

Deep learning methods to generate synthetic CT from MRI in radiotherapy: A literature review

PURPOSE

In radiotherapy, MRI is used for target volume and organs-at-risk delineation for its superior soft-tissue contrast as compared to CT imaging. However, MRI does not provide the electron density of tissue necessary for dose calculation. Several methods of synthetic-CT (sCT) generation from MRI data have been developed for radiotherapy dose calculation. This work reviewed deep learning (DL) sCT generation methods and their associated image and dose evaluation, in the context of MRI-based dose calculation.

METHODS

We searched the PubMed and ScienceDirect electronic databases from January 2010 to March 2021. For each paper, several items were screened and compiled in figures and tables.

RESULTS

This review included 57 studies. The DL methods were either generator-only based (45% of the reviewed studies), or generative adversarial network (GAN) architecture and its variants (55% of the reviewed studies). The brain and pelvis were the most commonly investigated anatomical localizations (39% and 28% of the reviewed studies, respectively), and more rarely, the head-and-neck (H&N) (15%), abdomen (10%), liver (5%) or breast (3%). All the studies performed an image evaluation of sCTs with a diversity of metrics, with only 36 studies performing dosimetric evaluations of sCT.

CONCLUSIONS

The median mean absolute errors were around 76 HU for the brain and H&N sCTs and 40 HU for the pelvis sCTs. For the brain, the mean dose difference between the sCT and the reference CT was <2%. For the H&N and pelvis, the mean dose difference was below 1% in most of the studies. Recent GAN architectures have advantages compared to generator-only, but no superiority was found in term of image or dose sCT uncertainties. Key challenges of DL-based sCT generation methods from MRI in radiotherapy is the management of movement for abdominal and thoracic localizations, the standardization of sCT evaluation, and the investigation of multicenter impacts.

Deep learning based synthetic-CT generation in radiotherapy and PET: A review

Recently,deep learning (DL)-based methods for the generation of synthetic computed tomography (sCT) have received significant research attention as an alternative to classical ones. We present here a systematic review of these methods by grouping them into three categories, according to their clinical applications: (i) to replace computed tomography in magnetic resonance (MR) based treatment planning, (ii) facilitate cone-beam computed tomography based image-guided adaptive radiotherapy, and (iii) derive attenuation maps for the correction of positron emission tomography. Appropriate database searching was performed on journal articles published between January 2014 and December 2020. The DL methods' key characteristics were extracted from each eligible study, and a comprehensive comparison among network architectures and metrics was reported. A detailed review of each category was given, highlighting essential contributions, identifying specific challenges, and summarizing the achievements. Lastly, the statistics of all the cited works from various aspects were analyzed, revealing the popularity and future trends and the potential of DL-based sCT generation. The current status of DL-based sCT generation was evaluated, assessing the clinical readiness of the presented methods.

Synthetic CT generation from weakly paired MR images using cycle-consistent GAN for MR-guided radiotherapy

Although MR-guided radiotherapy (MRgRT) is advancing rapidly, generating accurate synthetic CT (sCT) from MRI is still challenging. Previous approaches using deep neural networks require large dataset of precisely co-registered CT and MRI pairs that are difficult to obtain due to respiration and peristalsis. Here, we propose a method to generate sCT based on deep learning training with weakly paired CT and MR images acquired from an MRgRT system using a cycle-consistent GAN (CycleGAN) framework that allows the unpaired image-to-image translation in abdomen and thorax. Data from 90 cancer patients who underwent MRgRT were retrospectively used. CT images of the patients were aligned to the corresponding MR images using deformable registration, and the deformed CT (dCT) and MRI pairs were used for network training and testing. The 2.5D CycleGAN was constructed to generate sCT from the MRI input. To improve the sCT generation performance, a perceptual loss that explores the discrepancy between high-dimensional representations of images extracted from a well-trained classifier was incorporated into the CycleGAN. The CycleGAN with perceptual loss outperformed the U-net in terms of errors and similarities between sCT and dCT, and dose estimation for treatment planning of thorax, and abdomen. The sCT generated using CycleGAN produced virtually identical dose distribution maps and dose-volume histograms compared to dCT. CycleGAN with perceptual loss outperformed U-net in sCT generation when trained with weakly paired dCT-MRI for MRgRT. The proposed method will be useful to increase the treatment accuracy of MR-only or MR-guided adaptive radiotherapy.

Supplementary Information

The online version contains supplementary material available at 10.1007/s13534-021-00195-8.

An MR-only acquisition and artificial intelligence based image-processing protocol for photon and proton therapy using a low field MR

OBJECTIVE

Recent developments on synthetically generated CTs (sCT), hybrid MRI linacs and MR-only simulations underlined the clinical feasibility and acceptance of MR guided radiation therapy. However, considering clinical application of open and low field MR with a limited field of view can result in truncation of the patient's anatomy which further affects the MR to sCT conversion. In this study an acquisition protocol and subsequent MR image stitching is proposed to overcome the limited field of view restriction of open MR scanners, for MR-only photon and proton therapy.

MATERIAL AND METHODS

12 prostate cancer patients scanned with an open 0.35T scanner were included. To obtain the full body contour an enhanced imaging protocol including two repeated scans after bilateral table movement was introduced. All required structures (patient contour, target and organ at risk) were delineated on a post-processed combined transversal image set (stitched MRI). The postprocessed MR was converted into a sCT by a pretrained neural network generator. Inversely planned photon and proton plans (VMAT and SFUD) were designed using the sCT and recalculated for rigidly and deformably registered CT images and compared based on D2%, D50%, V70Gy for organs at risk and based on D2%, D50%, D98% for the CTV and PTV. The stitched MRI and the untruncated MRI were compared to the CT, and the maximum surface distance was calculated. The sCT was evaluated with respect to delineation accuracy by comparing on stitched MRI and sCT using the DICE coefficient for femoral bones and the whole body.

RESULTS

Maximum surface distance analysis revealed uncertainties in lateral direction of 1-3mm on average. DICE coefficient analysis confirms good performance of the sCT conversion, i.e. 92%, 93%, and 100% were obtained for femoral bone left and right and whole body. Dose comparison resulted in uncertainties below 1% between deformed CT and sCT and below 2% between rigidly registered CT and sCT in the CTV for photon and proton treatment plans.

DISCUSSION

A newly developed acquisition protocol for open MR scanners and subsequent Sct generation revealed good acceptance for photon and proton therapy. Moreover, this protocol tackles the restriction of the limited FOVs and expands the capacities towards MR guided proton therapy with horizontal beam lines.