Multicentre, deep learning, synthetic-CT generation for ano-rectal MR-only radiotherapy treatment planning

A unified generation-registration framework for improved MR-based CT synthesis in proton therapy

BACKGROUND

The use of magnetic resonance (MR) imaging for proton therapy treatment planning is gaining attention as a highly effective method for guidance. At the core of this approach is the generation of computed tomography (CT) images from MR scans. However, the critical issue in this process is accurately aligning the MR and CT images, a task that becomes particularly challenging in frequently moving body areas, such as the head-and-neck. Misalignments in these images can result in blurred synthetic CT (sCT) images, adversely affecting the precision and effectiveness of the treatment planning.

PURPOSE

This study introduces a novel network that cohesively unifies image generation and registration processes to enhance the quality and anatomical fidelity of sCTs derived from better-aligned MR images.

METHODS

The approach synergizes a generation network (G) with a deformable registration network (R), optimizing them jointly in MR-to-CT synthesis. This goal is achieved by alternately minimizing the discrepancies between the generated/registered CT images and their corresponding reference CT counterparts. The generation network employs a UNet architecture, while the registration network leverages an implicit neural representation (INR) of the displacement vector fields (DVFs). We validated this method on a dataset comprising 60 head-and-neck patients, reserving 12 cases for holdout testing.

RESULTS

Compared to the baseline Pix2Pix method with MAE 124.95 ± $\pm$ 30.74 HU, the proposed technique demonstrated 80.98 ± $\pm$ 7.55 HU. The unified translation-registration network produced sharper and more anatomically congruent outputs, showing superior efficacy in converting MR images to sCTs. Additionally, from a dosimetric perspective, the plan recalculated on the resulting sCTs resulted in a remarkably reduced discrepancy to the reference proton plans.

CONCLUSIONS

This study conclusively demonstrates that a holistic MR-based CT synthesis approach, integrating both image-to-image translation and deformable registration, significantly improves the precision and quality of sCT generation, particularly for the challenging body area with varied anatomic changes between corresponding MR and CT.

Challenges and opportunities in the development and clinical implementation of artificial intelligence based synthetic computed tomography for magnetic resonance only radiotherapy

Synthetic computed tomography (sCT) generated from magnetic resonance imaging (MRI) can serve as a substitute for planning CT in radiation therapy (RT), thereby removing registration uncertainties associated with multi-modality imaging pairing, reducing costs and patient radiation exposure. CE/FDA-approved sCT solutions are nowadays available for pelvis, brain, and head and neck, while more complex deep learning (DL) algorithms are under investigation for other anatomic sites. The main challenge in achieving a widespread clinical implementation of sCT lies in the absence of consensus on sCT commissioning and quality assurance (QA), resulting in variation of sCT approaches across different hospitals. To address this issue, a group of experts gathered at the ESTRO Physics Workshop 2022 to discuss the integration of sCT solutions into clinics and report the process and its outcomes. This position paper focuses on aspects of sCT development and commissioning, outlining key elements crucial for the safe implementation of an MRI-only RT workflow.

Synthetic CT generation for pelvic cases based on deep learning in multi-center datasets

BACKGROUND AND PURPOSE

To investigate the feasibility of synthesizing computed tomography (CT) images from magnetic resonance (MR) images in multi-center datasets using generative adversarial networks (GANs) for rectal cancer MR-only radiotherapy.

MATERIALS AND METHODS

Conventional T2-weighted MR and CT images were acquired from 90 rectal cancer patients at Peking University People's Hospital and 19 patients in public datasets. This study proposed a new model combining contrastive learning loss and consistency regularization loss to enhance the generalization of model for multi-center pelvic MRI-to-CT synthesis. The CT-to-sCT image similarity was evaluated by computing the mean absolute error (MAE), peak signal-to-noise ratio (SNRpeak), structural similarity index (SSIM) and Generalization Performance (GP). The dosimetric accuracy of synthetic CT was verified against CT-based dose distributions for the photon plan. Relative dose differences in the planning target volume and organs at risk were computed.

RESULTS

Our model presented excellent generalization with a GP of 0.911 on unseen datasets and outperformed the plain CycleGAN, where MAE decreased from 47.129 to 42.344, SNRpeak improved from 25.167 to 26.979, SSIM increased from 0.978 to 0.992. The dosimetric analysis demonstrated that most of the relative differences in dose and volume histogram (DVH) indicators between synthetic CT and real CT were less than 1%.

CONCLUSION

The proposed model can generate accurate synthetic CT in multi-center datasets from T2w-MR images. Most dosimetric differences were within clinically acceptable criteria for photon radiotherapy, demonstrating the feasibility of an MRI-only workflow for patients with rectal cancer.

3D Unsupervised deep learning method for magnetic resonance imaging-to-computed tomography synthesis in prostate radiotherapy

Background and purpose

Magnetic resonance imaging (MRI)-to-computed tomography (CT) synthesis is essential in MRI-only radiotherapy workflows, particularly through deep learning techniques known for their accuracy. However, current supervised methods are limited to specific center's learnings and depend on registration precision. The aim of this study was to evaluate the accuracy of unsupervised and supervised approaches in the context of prostate MRI-to-CT generation for radiotherapy dose calculation.

Methods

CT/MRI image pairs from 99 prostate cancer patients across three different centers were used. A comparison between supervised and unsupervised conditional Generative Adversarial Networks (cGAN) was conducted. Unsupervised training incorporates a style transfer method with. Content and Style Representation for Enhanced Perceptual synthesis (CREPs) loss. For dose evaluation, the photon prescription dose was 60 Gy delivered in volumetric modulated arc therapy (VMAT). Imaging endpoint for sCT evaluation was Mean Absolute Error (MAE). Dosimetric endpoints included absolute dose differences and gamma analysis between CT and sCT dose calculations.

Results

The unsupervised paired network exhibited the highest accuracy for the body with a MAE at 33.6 HU, the highest MAE was 45.5 HU obtained with unsupervised unpaired learning. All architectures provided clinically acceptable results for dose calculation with gamma pass rates above 94 % (1 % 1 mm 10 %).

Conclusions

This study shows that multicenter data can produce accurate sCTs via unsupervised learning, eliminating CT-MRI registration. The sCTs not only matched HU values but also enabled precise dose calculations, suggesting their potential for wider use in MRI-only radiotherapy workflows.

Artificial intelligence in radiotherapy: Current applications and future trends

Radiation therapy has dramatically changed with the advent of computed tomography and intensity modulation. This added complexity to the workflow but allowed for more precise and reproducible treatment. As a result, these advances required the accurate delineation of many more volumes, raising questions about how to delineate them, in a uniform manner across centers. Then, as computing power improved, reverse planning became possible and three-dimensional dose distributions could be generated. Artificial intelligence offers the opportunity to make such workflow more efficient while increasing practice homogeneity. Many artificial intelligence-based tools are being implemented in routine practice to increase efficiency, reduce workload and improve homogeneity of treatments. Data retrieved from this workflow could be combined with clinical data and omic data to develop predictive tools to support clinical decision-making process. Such predictive tools are at the stage of proof-of-concept and need to be explainatory, prospectively validated, and based on large and multicenter cohorts. Nevertheless, they could bridge the gap to personalized radiation oncology, by personalizing oncologic strategies, dose prescriptions to tumor volumes and dose constraints to organs at risk.

CycleSeg: Simultaneous synthetic CT generation and unsupervised segmentation for MR-only radiotherapy treatment planning of prostate cancer

BACKGROUND

MR-only radiotherapy treatment planning is an attractive alternative to conventional workflow, reducing scan time and ionizing radiation. It is crucial to derive the electron density map or synthetic CT (sCT) from MR data to perform dose calculations to enable MR-only treatment planning. Automatic segmentation of relevant organs in MR images can accelerate the process by preventing the time-consuming manual contouring step. However, the segmentation label is available only for CT data in many cases.

PURPOSE

We propose CycleSeg, a unified framework that generates sCT and corresponding segmentation from MR images without access to MR segmentation labels METHODS: CycleSeg utilizes the CycleGAN formulation to perform unpaired synthesis of sCT and image alignment. To enable MR (sCT) segmentation, CycleSeg incorporates unsupervised domain adaptation by using a pseudo-labeling approach with feature alignment in semantic segmentation space. In contrast to previous approaches that perform segmentation on MR data, CycleSeg could perform segmentation on both MR and sCT. Experiments were performed with data from prostate cancer patients, with 78/7/10 subjects in the training/validation/test sets, respectively.

RESULTS

CycleSeg showed the best sCT generation results, with the lowest mean absolute error of 102.2 and the lowest Fréchet inception distance of 13.0. CycleSeg also performed best on MR segmentation, with the highest average dice score of 81.0 and 81.1 for MR and sCT segmentation, respectively. Ablation experiments confirmed the contribution of the proposed components of CycleSeg.

CONCLUSION

CycleSeg effectively synthesized CT and performed segmentation on MR images of prostate cancer patients. Thus, CycleSeg has the potential to expedite MR-only radiotherapy treatment planning, reducing the prescribed scans and manual segmentation effort, and increasing throughput.

Magnetic resonance imaging-based bone imaging of the lower limb: Strategies for generating high-resolution synthetic computed tomography

This study aims at assessing approaches for generating high-resolution magnetic resonance imaging- (MRI-) based synthetic computed tomography (sCT) images suitable for orthopedic care using a deep learning model trained on low-resolution computed tomography (CT) data. To that end, paired MRI and CT data of three anatomical regions were used: high-resolution knee and ankle data, and low-resolution hip data. Four experiments were conducted to investigate the impact of low-resolution training CT data on sCT generation and to find ways to train models on low-resolution data while providing high-resolution sCT images. Experiments included resampling of the training data or augmentation of the low-resolution data with high-resolution data. Training sCT generation models using low-resolution CT data resulted in blurry sCT images. By resampling the MRI/CT pairs before the training, models generated sharper images, presumably through an increase in the MRI/CT mutual information. Alternatively, augmenting the low-resolution with high-resolution data improved sCT in terms of mean absolute error proportionally to the amount of high-resolution data. Overall, the morphological accuracy was satisfactory as assessed by an average intermodal distance between joint centers ranging from 0.7 to 1.2 mm and by an average intermodal root-mean-squared distances between bone surfaces under 0.7 mm. Average dice scores ranged from 79.8% to 87.3% for bony structures. To conclude, this paper proposed approaches to generate high-resolution sCT suitable for orthopedic care using low-resolution data. This can generalize the use of sCT for imaging the musculoskeletal system, paving the way for an MR-only imaging with simplified logistics and no ionizing radiation.

A systematic literature review: deep learning techniques for synthetic medical image generation and their applications in radiotherapy

The aim of this systematic review is to determine whether Deep Learning (DL) algorithms can provide a clinically feasible alternative to classic algorithms for synthetic Computer Tomography (sCT). The following categories are presented in this study: ∙ MR-based treatment planning and synthetic CT generation techniques. ∙ Generation of synthetic CT images based on Cone Beam CT images. ∙ Low-dose CT to High-dose CT generation. ∙ Attenuation correction for PET images. To perform appropriate database searches, we reviewed journal articles published between January 2018 and June 2023. Current methodology, study strategies, and results with relevant clinical applications were analyzed as we outlined the state-of-the-art of deep learning based approaches to inter-modality and intra-modality image synthesis. This was accomplished by contrasting the provided methodologies with traditional research approaches. The key contributions of each category were highlighted, specific challenges were identified, and accomplishments were summarized. As a final step, the statistics of all the cited works from various aspects were analyzed, which revealed that DL-based sCTs have achieved considerable popularity, while also showing the potential of this technology. In order to assess the clinical readiness of the presented methods, we examined the current status of DL-based sCT generation.

Artificial general intelligence for radiation oncology

The emergence of artificial general intelligence (AGI) is transforming radiation oncology. As prominent vanguards of AGI, large language models (LLMs) such as GPT-4 and PaLM 2 can process extensive texts and large vision models (LVMs) such as the Segment Anything Model (SAM) can process extensive imaging data to enhance the efficiency and precision of radiation therapy. This paper explores full-spectrum applications of AGI across radiation oncology including initial consultation, simulation, treatment planning, treatment delivery, treatment verification, and patient follow-up. The fusion of vision data with LLMs also creates powerful multimodal models that elucidate nuanced clinical patterns. Together, AGI promises to catalyze a shift towards data-driven, personalized radiation therapy. However, these models should complement human expertise and care. This paper provides an overview of how AGI can transform radiation oncology to elevate the standard of patient care in radiation oncology, with the key insight being AGI's ability to exploit multimodal clinical data at scale.

Quantitative assessment of adaptive radiotherapy for prostate cancer using deep learning: Bladder dose as a decision criterion

BACKGROUND

Adaptive radiotherapy (ART) can incorporate anatomical variations in a reoptimized treatment plan for fractionated radiotherapy. An automatic solution to objectively determine whether ART should be performed immediately after the daily image acquisition is highly desirable.

PURPOSE

We investigate a quantitative criterion for whether ART should be performed in prostate cancer radiotherapy by synthesizing pseudo-CT (sCT) images and evaluating dosimetric impact on treatment planning using deep learning approaches.

METHOD AND MATERIALS

Planning CT (pCT) and daily cone-beam CT (CBCT) data sets of 74 patients are used to train (60 patients) and evaluate (14 patients) a cycle adversarial generative network (CycleGAN) that performs the task of synthesizing high-quality sCT from daily CBCT. Automatic delineation (AD) of the bladder is performed on the sCT using the U-net. The combination of sCT and AD allows us to perform dose calculations based on the up-to-date bladder anatomy to determine whether the original treatment plan (ori-plan) is still applicable. For positive cases that the patients' anatomical changes and the associated dose calculations warrant re-planning, we made rapid plan revisions (re-plan) based on the ori-plan.

RESULTS

The mean absolute error within the region-of-interests (i.e., body, bladder, fat, muscle) between the sCT and pCT are 41.2, 25.1, 26.5, and 29.0HU, respectively. Taking the calculated results of pCT doses as the standard, for PTV, the gamma passing rates of sCT doses at 1 mm/1%, 2 mm/2% are 87.92%, 98.78%, respectively. The Dice coefficients of the AD-contours are 0.93 on pCT and 0.91 on sCT. According to the result of dose calculation, we found when the bladder volume underwent a substantial change (79.7%), the bladder dose is still within the safe limit, suggesting it is insufficient to solely use the bladder volume change as a criterion to determine whether adaptive treatment needs to be done. After AD-contours of the bladder using sCT, there are two cases whose bladder doseD mean > 4000 cGy ${{\mathrm{D}}}_{{\mathrm{mean}}} > 4000{\mathrm{\ cGy}}$ . For the two cases, we perform re-planning to reduce the bladder dose toD mean = 3841 cGy ${{\mathrm{D}}}_{{\mathrm{mean}}} = 3841{\mathrm{\ cGy}}$ ,D mean = 3580 cGy ${{\mathrm{D}}}_{{\mathrm{mean}}} = 3580{\mathrm{\ cGy\ }}$ under the condition that the PTV meets the prescribed dose.

CONCLUSION

We provide a dose accurate adaptive workflow for prostate cancer patients by using deep learning approaches, and implement ART that adapts to bladder dose. Of note, the specific replanning criterion for whether ART needs to be performed can adapt to different centers' choices based on their experience and daily observations.

Computed tomography synthesis from magnetic resonance imaging using cycle Generative Adversarial Networks with multicenter learning

Background and Purpose: Addressing the need for accurate dose calculation in MRI-only radiotherapy, the generation of synthetic Computed Tomography (sCT) from MRI has emerged. Deep learning (DL) techniques, have shown promising results in achieving high sCT accuracies. However, existing sCT synthesis methods are often center-specific, posing a challenge to their generalizability. To overcome this limitation, recent studies have proposed approaches, such as multicenter training . Material and methods: The purpose of this work was to propose a multicenter sCT synthesis by DL, using a 2D cycle-GAN on 128 prostate cancer patients, from four different centers. Four cases were compared: monocenter cases, monocenter training and test on another center, multicenter trainings and a test on a center not included in the training and multicenter trainings with an included center in the test. Trainings were performed using 20 patients. sCT accuracy evaluation was performed using Mean Absolute Error, Mean Error and Peak-Signal-to-Noise-Ratio. Dose accuracy was assessed with gamma index and Dose Volume Histogram comparison. Results: Qualitative, quantitative and dose results show that the accuracy of sCTs for monocenter trainings and multicenter trainings using a seen center in the test did not differ significantly. However, when the test involved an unseen center, the sCT quality was inferior. Conclusions: The aim of this work was to propose generalizable multicenter training for MR-to-CT synthesis. It was shown that only a few data from one center included in the training cohort allows sCT accuracy equivalent to a monocenter study.

Comprehensive dose evaluation of a Deep Learning based synthetic Computed Tomography algorithm for pelvic Magnetic Resonance-only radiotherapy

BACKGROUND AND PURPOSE

Magnetic Resonance (MR)-only radiotherapy enables the use of MR without the uncertainty of MR-Computed Tomography (CT) registration. This requires a synthetic CT (sCT) for dose calculations, which can be facilitated by a novel Zero Echo Time (ZTE) sequence where bones are visible and images are acquired in 65 seconds. This study evaluated the dose calculation accuracy for pelvic sites of a ZTE-based Deep Learning sCT algorithm developed by GE Healthcare.

MATERIALS AND METHODS

ZTE and CT images were acquired in 56 pelvic radiotherapy patients in the radiotherapy position. A 2D U-net convolutional neural network was trained using pairs of deformably registered CT and ZTE images from 36 patients. In the remaining 20 patients the dosimetric accuracy of the sCT was assessed using cylindrical dummy Planning Target Volumes (PTVs) positioned at four different central axial locations, as well as the clinical treatment plans (for prostate (n=10), rectum (n=4) and anus (n=6) cancers). The sCT was rigidly and deformably registered, the plan recalculated and the doses compared using mean differences and gamma analysis.

RESULTS

Mean dose differences to the PTV D98% were ≤ 0.5% for all dummy PTVs and clinical plans (rigid registration). Mean gamma pass rates at 1%/1 mm were 98.0 ± 0.4% (rigid) and 100.0 ± 0.0% (deformable), 96.5 ± 0.8% and 99.8 ± 0.1%, and 95.4 ± 0.6% and 99.4 ± 0.4% for the clinical prostate, rectum and anus plans respectively.

CONCLUSIONS

A ZTE-based sCT algorithm with high dose accuracy throughout the pelvis has been developed. This suggests the algorithm is sufficiently accurate for MR-only radiotherapy for all pelvic sites.

A boundary-guided transformer for measuring distance from rectal tumor to anal verge on magnetic resonance images

Accurate measurement of the distance from the tumor's lowest boundary to the anal verge (DTAV) provides an important reference value for treatment of rectal cancer, but the standard measurement method (colonoscopy) causes substantial pain. Therefore, we propose a method for automatically measuring the DTAV on sagittal magnetic resonance (MR) images. We designed a boundary-guided transformer that can accurately segment the rectum and tumor. From the segmentation results, we estimated the DTAV by automatically extracting the anterior rectal wall from the tumor's lowest point to the anal verge and then calculating its physical length. Experiments were conducted on a rectal tumor MR imaging (MRI) dataset to evaluate the efficacy of our method. The results showed that our method outperformed surgeons with 6 years of experience (p < 0.001). Furthermore, by referring to our segmentation results, attending and resident surgeons could improve their measurement precision and efficiency.

Deep Learning in MRI-guided Radiation Therapy: A Systematic Review

MRI-guided radiation therapy (MRgRT) offers a precise and adaptive approach to treatment planning. Deep learning applications which augment the capabilities of MRgRT are systematically reviewed. MRI-guided radiation therapy offers a precise, adaptive approach to treatment planning. Deep learning applications which augment the capabilities of MRgRT are systematically reviewed with emphasis placed on underlying methods. Studies are further categorized into the areas of segmentation, synthesis, radiomics, and real time MRI. Finally, clinical implications, current challenges, and future directions are discussed.

Measuring distance from lowest boundary of rectal tumor to anal verge on CT images using pyramid attention pooling transformer

Accurately measuring the Distance from the lowest boundary of rectal tumor To the Anal Verge (DTAV) is critical for developing optimal surgical plans for treating patients with rectal cancer. DTAV was traditionally estimated by colonoscopy or manual measurement on computed tomography (CT) images. However, colonoscopy brings substantial pains to the patient. As for manual measurement on CT images, it is time-consuming and its accuracy depends on the surgeon's expertise. In this work, we present a novel method for automatically measuring DTAV from sagittal CT images. The success of our method is mainly credited to a pyramid attention pooling (PAP) transformer architecture, which naturally entangles global lesion localization and local boundary delineation. Our method automatically generates the rectum's centerline based on a segmented rectum and tumor image to simulate the manual measurement of DTAV. We conduct a comprehensive evaluation of the method with a newly collected rectum tumor CT image dataset. On a test dataset of 48 patients' CT images with rectal tumors, the mean absolute difference between our method and the gold standard is 1.74 cm, which is a significant improvement of 1.29 cm over that measured by a resident surgeon (P < 0.001). In addition, The results measured by the resident surgeon referring to our segmentation results improved by 1.46 cm compared to the results measured independently by the residents. As experimentally demonstrated, our method exhibits great application potential in clinical scenarios.

Ensemble learning and personalized training for the improvement of unsupervised deep learning-based synthetic CT reconstruction

BACKGROUND

The growing adoption of magnetic resonance imaging (MRI)-guided radiation therapy (RT) platforms and a focus on MRI-only RT workflows have brought the technical challenge of synthetic computed tomography (sCT) reconstruction to the forefront. Unpaired-data deep learning-based approaches to the problem offer the attractive characteristic of not requiring paired training data, but the gap between paired- and unpaired-data results can be limiting.

PURPOSE

We present two distinct approaches aimed at improving unpaired-data sCT reconstruction results: a cascade ensemble that combines multiple models and a personalized training strategy originally designed for the paired-data setting.

METHODS

Comparisons are made between the following models: (1) the paired-data fully convolutional DenseNet (FCDN), (2) the FCDN with the Intentional Deep Overfit Learning (IDOL) personalized training strategy, (3) the unpaired-data CycleGAN, (4) the CycleGAN with the IDOL training strategy, and (5) the CycleGAN as an intermediate model in a cascade ensemble approach. Evaluation of the various models over 25 total patients is carried out using a five-fold cross-validation scheme, with the patient-specific IDOL models being trained for the five patients of fold 3, chosen at random.

RESULTS

In both the paired- and unpaired-data settings, adopting the IDOL training strategy led to improvements in the mean absolute error (MAE) between true CT images and sCT outputs within the body contour (mean improvement, paired- and unpaired-data approaches, respectively: 38%, 9%) and in regions of bone (52%, 5%), the peak signal-to-noise ratio (PSNR; 15%, 7%), and the structural similarity index (SSIM; 6%, <1%). The ensemble approach offered additional benefits over the IDOL approach in all three metrics (mean improvement over unpaired-data approach in fold 3; MAE: 20%; bone MAE: 16%; PSNR: 10%; SSIM: 2%), and differences in body MAE between the ensemble approach and the paired-data approach are statistically insignificant.

CONCLUSIONS

We have demonstrated that both a cascade ensemble approach and a personalized training strategy designed initially for the paired-data setting offer significant improvements in image quality metrics for the unpaired-data sCT reconstruction task. Closing the gap between paired- and unpaired-data approaches is a step toward fully enabling these powerful and attractive unpaired-data frameworks.

CT synthesis from MR in the pelvic area using Residual Transformer Conditional GAN

Magnetic resonance (MR) image-guided radiation therapy is a hot topic in current radiation therapy research, which relies on MR to generate synthetic computed tomography (SCT) images for radiation therapy. Convolution-based generative adversarial networks (GAN) have achieved promising results in synthesizing CT from MR since the introduction of deep learning techniques. However, due to the local limitations of pure convolutional neural networks (CNN) structure and the local mismatch between paired MR and CT images, particularly in pelvic soft tissue, the performance of GAN in synthesizing CT from MR requires further improvement. In this paper, we propose a new GAN called Residual Transformer Conditional GAN (RTCGAN), which exploits the advantages of CNN in local texture details and Transformer in global correlation to extract multi-level features from MR and CT images. Furthermore, the feature reconstruction loss is used to further constrain the image potential features, reducing over-smoothing and local distortion of the SCT. The experiments show that RTCGAN is visually closer to the reference CT (RCT) image and achieves desirable results on local mismatch tissues. In the quantitative evaluation, the MAE, SSIM, and PSNR of RTCGAN are 45.05 HU, 0.9105, and 28.31 dB, respectively. All of them outperform other comparison methods, such as deep convolutional neural networks (DCNN), Pix2Pix, Attention-UNet, WPD-DAGAN, and HDL.

Synthetic computed tomography generation for abdominal adaptive radiotherapy using low-field magnetic resonance imaging

Background and Purpose

Magnetic Resonance guided Radiotherapy (MRgRT) still needs the acquisition of Computed Tomography (CT) images and co-registration between CT and Magnetic Resonance Imaging (MRI). The generation of synthetic CT (sCT) images from the MR data can overcome this limitation. In this study we aim to propose a Deep Learning (DL) based approach for sCT image generation for abdominal Radiotherapy using low field MR images.

Materials and methods

CT and MR images were collected from 76 patients treated on abdominal sites. U-Net and conditional Generative Adversarial Network (cGAN) architectures were used to generate sCT images. Additionally, sCT images composed of only six bulk densities were generated with the aim of having a Simplified sCT.Radiotherapy plans calculated using the generated images were compared to the original plan in terms of gamma pass rate and Dose Volume Histogram (DVH) parameters.

Results

sCT images were generated in 2 s and 2.5 s with U-Net and cGAN architectures respectively.Gamma pass rates for 2%/2mm and 3%/3mm criteria were 91% and 95% respectively. Dose differences within 1% for DVH parameters on the target volume and organs at risk were obtained.

Conclusion

U-Net and cGAN architectures are able to generate abdominal sCT images fast and accurately from low field MRI.

A deep learning approach to generate synthetic CT in low field MR-guided radiotherapy for lung cases

INTRODUCTION

This study aims to apply a conditional Generative Adversarial Network (cGAN) to generate synthetic Computed Tomography (sCT) from 0.35 Tesla Magnetic Resonance (MR) images of the thorax.

METHODS

Sixty patients treated for lung lesions were enrolled and divided into training (32), validation (8), internal (10,T_A) and external (10,T_B) test set. Image accuracy of generated sCT was evaluated computing the mean absolute (MAE) and mean error (ME) with respect the original CT. Three treatment plans were calculated for each patient considering MRI as reference image: original CT, sCT (pure sCT) and sCT with GTV density override (hybrid sCT) were used as Electron Density (ED) map. Dose accuracy was evaluated comparing treatment plans in terms of gamma analysis and Dose Volume Histogram (DVH) parameters.

RESULTS

No significant difference was observed between the test sets for image and dose accuracy parameters. Considering the whole test cohort, a MAE of 54.9 ± 10.5 HU and a ME of 4.4 ± 7.4 HU was obtained. Mean gamma passing rates for 2%/2mm, and 3%/3mm tolerance criteria were 95.5 ± 5.9% and 98.2 ± 4.1% for pure sCT, 96.1 ± 5.1% and 98.5 ± 3.9% for hybrid sCT: the difference between the two approaches was significant (p = 0.01). As regards DVH analysis, differences in target parameters estimation were found to be within 5% using hybrid approach and 20% using pure sCT.

CONCLUSION

The DL algorithm here presented can generate sCT images in the thorax with good image and dose accuracy, especially when the hybrid approach is used. The algorithm does not suffer from inter-scanner variability, making feasible the implementation of MR-only workflows for palliative treatments.

Quality assurance for MRI-only radiation therapy: A voxel-wise population-based methodology for image and dose assessment of synthetic CT generation methods

The quality assurance of synthetic CT (sCT) is crucial for safe clinical transfer to an MRI-only radiotherapy planning workflow. The aim of this work is to propose a population-based process assessing local errors in the generation of sCTs and their impact on dose distribution. For the analysis to be anatomically meaningful, a customized interpatient registration method brought the population data to the same coordinate system. Then, the voxel-based process was applied on two sCT generation methods: a bulk-density method and a generative adversarial network. The CT and MRI pairs of 39 patients treated by radiotherapy for prostate cancer were used for sCT generation, and 26 of them with delineated structures were selected for analysis. Voxel-wise errors in sCT compared to CT were assessed for image intensities and dose calculation, and a population-based statistical test was applied to identify the regions where discrepancies were significant. The cumulative histograms of the mean absolute dose error per volume of tissue were computed to give a quantitative indication of the error for each generation method. Accurate interpatient registration was achieved, with mean Dice scores higher than 0.91 for all organs. The proposed method produces three-dimensional maps that precisely show the location of the major discrepancies for both sCT generation methods, highlighting the heterogeneity of image and dose errors for sCT generation methods from MRI across the pelvic anatomy. Hence, this method provides additional information that will assist with both sCT development and quality control for MRI-based planning radiotherapy.

Synthetic CT generation for MRI-guided adaptive radiotherapy in prostate cancer

Current MRI-guided adaptive radiotherapy (MRgART) workflows require fraction-specific electron and/or mass density maps, which are created by deformable image registration (DIR) between the simulation CT images and daily MR images. Manual density overrides may also be needed where DIR-produced results are inaccurate. This approach slows the adaptive radiotherapy workflow and introduces additional dosimetric uncertainties, especially in the presence of the magnetic field. This study investigated a method based on a conditional generative adversarial network (cGAN) with a multi-planar method to generate synthetic CT images from low-field MR images to improve efficiency in MRgART workflows for prostate cancer. Fifty-seven male patients, who received MRI-guided radiation therapy to the pelvis using the ViewRay MRIdian Linac, were selected. Forty-five cases were randomly assigned to the training cohort with the remaining twelve cases assigned to the validation/testing cohort. All patient datasets had a semi-paired DIR-deformed CT-sim image and 0.35T MR image acquired using a true fast imaging with steady-state precession (TrueFISP) sequence. Synthetic CT images were compared with deformed CT images to evaluate image quality and dosimetric accuracy. To evaluate the dosimetric accuracy of this method, clinical plans were recalculated on synthetic CT images in the MRIdian treatment planning system. Dose volume histograms for planning target volumes (PTVs) and organs-at-risk (OARs) and dose distributions using gamma analyses were evaluated. The mean-absolute-errors (MAEs) in CT numbers were 30.1 ± 4.2 HU, 19.6 ± 2.3 HU and 158.5 ± 26.0 HU for the whole pelvis, soft tissue, and bone, respectively. The peak signal-to-noise ratio was 35.2 ± 1.7 and the structural index similarity measure was 0.9758 ± 0.0035. The dosimetric difference was on average less than 1% for all PTV and OAR metrics. Plans showed good agreement with gamma pass rates of 99% and 99.9% for 1%/1 mm and 2%/2 mm, respectively. Our study demonstrates the potential of using synthetic CT images created with a multi-planar cGAN method from 0.35T MRI TrueFISP images for the MRgART treatment of prostate radiotherapy. Future work will validate the method in a large cohort of patients and investigate the limitations of the method in the adaptive workflow.

A Survey on Deep Learning for Precision Oncology

Precision oncology, which ensures optimized cancer treatment tailored to the unique biology of a patient’s disease, has rapidly developed and is of great clinical importance. Deep learning has become the main method for precision oncology. This paper summarizes the recent deep-learning approaches relevant to precision oncology and reviews over 150 articles within the last six years. First, we survey the deep-learning approaches categorized by various precision oncology tasks, including the estimation of dose distribution for treatment planning, survival analysis and risk estimation after treatment, prediction of treatment response, and patient selection for treatment planning. Secondly, we provide an overview of the studies per anatomical area, including the brain, bladder, breast, bone, cervix, esophagus, gastric, head and neck, kidneys, liver, lung, pancreas, pelvis, prostate, and rectum. Finally, we highlight the challenges and discuss potential solutions for future research directions.

Towards a safe and efficient clinical implementation of machine learning in radiation oncology by exploring model interpretability, explainability and data-model dependency

The interest in machine learning (ML) has grown tremendously in recent years, partly due to the performance leap that occurred with new techniques of deep learning, convolutional neural networks for images, increased computational power, and wider availability of large datasets. Most fields of medicine follow that popular trend and, notably, radiation oncology is one of those that are at the forefront, with already a long tradition in using digital images and fully computerized workflows. ML models are driven by data, and in contrast with many statistical or physical models, they can be very large and complex, with countless generic parameters. This inevitably raises two questions, namely, the tight dependence between the models and the datasets that feed them, and the interpretability of the models, which scales with its complexity. Any problems in the data used to train the model will be later reflected in their performance. This, together with the low interpretability of ML models, makes their implementation into the clinical workflow particularly difficult. Building tools for risk assessment and quality assurance of ML models must involve then two main points: interpretability and data-model dependency. After a joint introduction of both radiation oncology and ML, this paper reviews the main risks and current solutions when applying the latter to workflows in the former. Risks associated with data and models, as well as their interaction, are detailed. Next, the core concepts of interpretability, explainability, and data-model dependency are formally defined and illustrated with examples. Afterwards, a broad discussion goes through key applications of ML in workflows of radiation oncology as well as vendors' perspectives for the clinical implementation of ML.

Comparison of Synthetic Computed Tomography Generation Methods, Incorporating Male and Female Anatomical Differences, for Magnetic Resonance Imaging-Only Definitive Pelvic Radiotherapy

Purpose

There are several means of synthetic computed tomography (sCT) generation for magnetic resonance imaging (MRI)-only planning; however, much of the research omits large pelvic treatment regions and female anatomical specific methods. This research aimed to apply four of the most popular methods of sCT creation to facilitate MRI-only radiotherapy treatment planning for male and female anorectal and gynecological neoplasms. sCT methods were validated against conventional computed tomography (CT), with regard to Hounsfield unit (HU) estimation and plan dosimetry.

Methods and Materials

Paired MRI and CT scans of 40 patients were used for sCT generation and validation. Bulk density assignment, tissue class density assignment, hybrid atlas, and deep learning sCT generation methods were applied to all 40 patients. Dosimetric accuracy was assessed by dose difference at reference point, dose volume histogram (DVH) parameters, and 3D gamma dose comparison. HU estimation was assessed by mean error and mean absolute error in HU value between each sCT and CT.

Results

The median percentage dose difference between the CT and sCT was <1.0% for all sCT methods. The deep learning method resulted in the lowest median percentage dose difference to CT at −0.03% (IQR 0.13, −0.31) and bulk density assignment resulted in the greatest difference at −0.73% (IQR −0.10, −1.01). The mean 3D gamma dose agreement at 3%/2 mm among all sCT methods was 99.8%. The highest agreement at 1%/1 mm was 97.3% for the deep learning method and the lowest was 93.6% for the bulk density method. Deep learning and hybrid atlas techniques gave the lowest difference to CT in mean error and mean absolute error in HU estimation.

Conclusions

All methods of sCT generation used in this study resulted in similarly high dosimetric agreement for MRI-only planning of male and female cancer pelvic regions. The choice of the sCT generation technique can be guided by department resources available and image guidance considerations, with minimal impact on dosimetric accuracy.

Deep learning-based automatic delineation of anal cancer gross tumour volume: a multimodality comparison of CT, PET and MRI

BACKGROUND

Accurate target volume delineation is a prerequisite for high-precision radiotherapy. However, manual delineation is resource-demanding and prone to interobserver variation. An automatic delineation approach could potentially save time and increase delineation consistency. In this study, the applicability of deep learning for fully automatic delineation of the gross tumour volume (GTV) in patients with anal squamous cell carcinoma (ASCC) was evaluated for the first time. An extensive comparison of the effects single modality and multimodality combinations of computed tomography (CT), positron emission tomography (PET), and magnetic resonance imaging (MRI) have on automatic delineation quality was conducted.

MATERIAL AND METHODS

18F-fluorodeoxyglucose PET/CT and contrast-enhanced CT (ceCT) images were collected for 86 patients with ASCC. A subset of 36 patients also underwent a study-specific 3T MRI examination including T2- and diffusion-weighted imaging. The resulting two datasets were analysed separately. A two-dimensional U-Net convolutional neural network (CNN) was trained to delineate the GTV in axial image slices based on single or multimodality image input. Manual GTV delineations constituted the ground truth for CNN model training and evaluation. Models were evaluated using the Dice similarity coefficient (Dice) and surface distance metrics computed from five-fold cross-validation.

RESULTS

CNN-generated automatic delineations demonstrated good agreement with the ground truth, resulting in mean Dice scores of 0.65-0.76 and 0.74-0.83 for the 86 and 36-patient datasets, respectively. For both datasets, the highest mean Dice scores were obtained using a multimodal combination of PET and ceCT (0.76-0.83). However, models based on single modality ceCT performed comparably well (0.74-0.81). T2W-only models performed acceptably but were somewhat inferior to the PET/ceCT and ceCT-based models.

CONCLUSION

CNNs provided high-quality automatic GTV delineations for both single and multimodality image input, indicating that deep learning may prove a versatile tool for target volume delineation in future patients with ASCC.

Feasibility of Synthetic Computed Tomography Images Generated from Magnetic Resonance Imaging Scans Using Various Deep Learning Methods in the Planning of Radiation Therapy for Prostate Cancer

Simple Summary

MRI-only simulation in radiation therapy (RT) planning has received attention because the CT scan can be omitted. For MRI-only simulation, synthetic CT (sCT) is necessary for the dose calculation. Various methodologies have been suggested for the generation of sCT and, recently, methods using the deep learning approaches are actively investigated. GAN and cycle-consistent GAN (CycGAN) have been mainly tested, however, very limited studies compared the qualities of sCTs generated from these methods or suggested other models for sCT generation. We have compared GAN, CycGAN, and, reference-guided GAN (RgGAN), a new model of deep learning method. We found that the performance in the HU conservation for soft tissue was poorest for GAN. All methods could generate sCTs feasible for VMAT planning with the trend that sCT generated from the RgGAN showed best performance in dosimetric conservation D_98% and D_95% than sCTs from other methodologies.

Abstract

We aimed to evaluate and compare the qualities of synthetic computed tomography (sCT) generated by various deep-learning methods in volumetric modulated arc therapy (VMAT) planning for prostate cancer. Simulation computed tomography (CT) and T2-weighted simulation magnetic resonance image from 113 patients were used in the sCT generation by three deep-learning approaches: generative adversarial network (GAN), cycle-consistent GAN (CycGAN), and reference-guided CycGAN (RgGAN), a new model which performed further adjustment of sCTs generated by CycGAN with available paired images. VMAT plans on the original simulation CT images were recalculated on the sCTs and the dosimetric differences were evaluated. For soft tissue, a significant difference in the mean Hounsfield unites (HUs) was observed between the original CT images and only sCTs from GAN (p = 0.03). The mean relative dose differences for planning target volumes or organs at risk were within 2% among the sCTs from the three deep-learning approaches. The differences in dosimetric parameters for D_98% and D_95% from original CT were lowest in sCT from RgGAN. In conclusion, HU conservation for soft tissue was poorest for GAN. There was the trend that sCT generated from the RgGAN showed best performance in dosimetric conservation D_98% and D_95% than sCTs from other methodologies.

The benefit of MR-only radiotherapy treatment planning for anal and rectal cancers: A planning study

INTRODUCTION

Limited evidence exists showing the benefit of magnetic resonance (MR)-only radiotherapy treatment planning for anal and rectal cancers. This study aims to assess the impact of MR-only planning on target volumes (TVs) and treatment plan doses to organs at risks (OARs) for anal and rectal cancers versus a computed tomography (CT)-only pathway.

MATERIALS AND METHODS

Forty-six patients (29 rectum and 17 anus) undergoing preoperative or radical external beam radiotherapy received CT and T2 MR simulation. TV and OARs were delineated on CT and MR, and volumetric arc therapy treatment plans were optimized independently (53.2 Gy/28 fractions for anus, 45 Gy/25 fractions for rectum). Further treatment plans assessed gross tumor volume (GTV) dose escalation. Differences in TV volumes and OAR doses, in terms of Vx Gy (organ volume (%) receiving x dose (Gy)), were assessed.

RESULTS

MR GTV and primary planning TV (PTV) volumes systematically reduced by 13 cc and 98 cc (anus) and 44 cc and 109 cc (rectum) respectively compared to CT volumes. Statistically significant OAR dose reductions versus CT were found for bladder and uterus (rectum) and bladder, penile bulb, and genitalia (anus). With GTV boosting, statistically significant dose reductions were found for sigmoid, small bowel, vagina, and penile bulb (rectum) and vagina (anus).

CONCLUSION

Our findings provide evidence that the introduction of MR (whether through MR-only or CT-MR pathways) to radiotherapy treatment planning for anal and rectal cancers has the potential to improve treatments. MR-related OAR dose reductions may translate into less treatment-related toxicity for patients or greater ability to dose escalate.

Artificial intelligence for assisting cancer diagnosis and treatment in the era of precision medicine

Over the past decade, artificial intelligence (AI) has contributed substantially to the resolution of various medical problems, including cancer. Deep learning (DL), a subfield of AI, is characterized by its ability to perform automated feature extraction and has great power in the assimilation and evaluation of large amounts of complicated data. On the basis of a large quantity of medical data and novel computational technologies, AI, especially DL, has been applied in various aspects of oncology research and has the potential to enhance cancer diagnosis and treatment. These applications range from early cancer detection, diagnosis, classification and grading, molecular characterization of tumors, prediction of patient outcomes and treatment responses, personalized treatment, automatic radiotherapy workflows, novel anti-cancer drug discovery, and clinical trials. In this review, we introduced the general principle of AI, summarized major areas of its application for cancer diagnosis and treatment, and discussed its future directions and remaining challenges. As the adoption of AI in clinical use is increasing, we anticipate the arrival of AI-powered cancer care.

Deep learning methods to generate synthetic CT from MRI in radiotherapy: A literature review

PURPOSE

In radiotherapy, MRI is used for target volume and organs-at-risk delineation for its superior soft-tissue contrast as compared to CT imaging. However, MRI does not provide the electron density of tissue necessary for dose calculation. Several methods of synthetic-CT (sCT) generation from MRI data have been developed for radiotherapy dose calculation. This work reviewed deep learning (DL) sCT generation methods and their associated image and dose evaluation, in the context of MRI-based dose calculation.

METHODS

We searched the PubMed and ScienceDirect electronic databases from January 2010 to March 2021. For each paper, several items were screened and compiled in figures and tables.

RESULTS

This review included 57 studies. The DL methods were either generator-only based (45% of the reviewed studies), or generative adversarial network (GAN) architecture and its variants (55% of the reviewed studies). The brain and pelvis were the most commonly investigated anatomical localizations (39% and 28% of the reviewed studies, respectively), and more rarely, the head-and-neck (H&N) (15%), abdomen (10%), liver (5%) or breast (3%). All the studies performed an image evaluation of sCTs with a diversity of metrics, with only 36 studies performing dosimetric evaluations of sCT.

CONCLUSIONS

The median mean absolute errors were around 76 HU for the brain and H&N sCTs and 40 HU for the pelvis sCTs. For the brain, the mean dose difference between the sCT and the reference CT was <2%. For the H&N and pelvis, the mean dose difference was below 1% in most of the studies. Recent GAN architectures have advantages compared to generator-only, but no superiority was found in term of image or dose sCT uncertainties. Key challenges of DL-based sCT generation methods from MRI in radiotherapy is the management of movement for abdominal and thoracic localizations, the standardization of sCT evaluation, and the investigation of multicenter impacts.

Deep learning based synthetic-CT generation in radiotherapy and PET: A review

Recently,deep learning (DL)-based methods for the generation of synthetic computed tomography (sCT) have received significant research attention as an alternative to classical ones. We present here a systematic review of these methods by grouping them into three categories, according to their clinical applications: (i) to replace computed tomography in magnetic resonance (MR) based treatment planning, (ii) facilitate cone-beam computed tomography based image-guided adaptive radiotherapy, and (iii) derive attenuation maps for the correction of positron emission tomography. Appropriate database searching was performed on journal articles published between January 2014 and December 2020. The DL methods' key characteristics were extracted from each eligible study, and a comprehensive comparison among network architectures and metrics was reported. A detailed review of each category was given, highlighting essential contributions, identifying specific challenges, and summarizing the achievements. Lastly, the statistics of all the cited works from various aspects were analyzed, revealing the popularity and future trends and the potential of DL-based sCT generation. The current status of DL-based sCT generation was evaluated, assessing the clinical readiness of the presented methods.

Patient position verification in magnetic-resonance imaging only radiotherapy of anal and rectal cancers

BACKGROUND AND PURPOSE

Magnetic resonance (MR)-only treatment pathways require either the MR-simulation or synthetic-computed tomography (sCT) as an alternative reference image for cone beam computed tomography (CBCT) patient position verification. This study assessed whether using T2 MR or sCT as CBCT reference images introduces systematic registration errors as compared to CT for anal and rectal cancers.

MATERIALS AND METHODS

A total of 32 patients (18 rectum,14 anus) received pre-treatment CT- and T2 MR- simulation. Routine treatment CBCTs were acquired. sCTs were generated using a validated research model. The local clinical registration protocol, using a grey-scale registration algorithm, was performed for 216 CBCTs using CT, MR and sCT as the reference image. Linear mixed effects modelling identified systematic differences between modalities.

RESULTS

Systematic translation and rotation differences to CT for MR were -0.3 to + 0.3 mm and -0.1 to 0.4° for anal cancers and -0.4 to 0.0 mm and 0.0 to 0.1° for rectal cancers, and for sCT were -0.4 to + 0.8 mm, -0.1 to 0.2° for anal cancers and -0.6 to + 0.2 mm, -0.1 to + 0.1° for rectal cancers.

CONCLUSIONS

T2 MR or sCT can successfully be used as reference images for anal and rectal cancer CBCT position verification with systematic differences to CT <±1 mm and <±0.5°. Clinical enabling of alternative modalities as reference images by vendors is required to reduce challenges associated with their use.

Artificial Intelligence in magnetic Resonance guided Radiotherapy: Medical and physical considerations on state of art and future perspectives

Over the last years, technological innovation in Radiotherapy (RT) led to the introduction of Magnetic Resonance-guided RT (MRgRT) systems. Due to the higher soft tissue contrast compared to on-board CT-based systems, MRgRT is expected to significantly improve the treatment in many situations. MRgRT systems may extend the management of inter- and intra-fraction anatomical changes, offering the possibility of online adaptation of the dose distribution according to daily patient anatomy and to directly monitor tumor motion during treatment delivery by means of a continuous cine MR acquisition. Online adaptive treatments require a multidisciplinary and well-trained team, able to perform a series of operations in a safe, precise and fast manner while the patient is waiting on the treatment couch. Artificial Intelligence (AI) is expected to rapidly contribute to MRgRT, primarily by safely and efficiently automatising the various manual operations characterizing online adaptive treatments. Furthermore, AI is finding relevant applications in MRgRT in the fields of image segmentation, synthetic CT reconstruction, automatic (on-line) planning and the development of predictive models based on daily MRI. This review provides a comprehensive overview of the current AI integration in MRgRT from a medical physicist's perspective. Medical physicists are expected to be major actors in solving new tasks and in taking new responsibilities: their traditional role of guardians of the new technology implementation will change with increasing emphasis on the managing of AI tools, processes and advanced systems for imaging and data analysis, gradually replacing many repetitive manual tasks.