Synthetic CT generation from MRI using 3D transformer-based denoising diffusion model

Feasibility of generating sagittal radiographs from coronal views using GAN-based deep learning framework in adolescent idiopathic scoliosis

BACKGROUND

Minimizing radiation exposure is crucial in monitoring adolescent idiopathic scoliosis (AIS). Generative adversarial networks (GANs) have emerged as valuable tools being able to generate high-quality synthetic images. This study explores the use of GANs to generate synthetic sagittal radiographs from coronal views in AIS patients.

METHODS

A dataset of 3,935 AIS patients who underwent spine and pelvis radiographic examinations using the EOS system, which simultaneously acquires coronal and sagittal images, was analyzed. The dataset was divided into training-set (85%, n = 3,356) and test-set (15%, n = 579). GAN model was trained to generate sagittal images from coronal views, with real sagittal views as reference standard. To assess accuracy, 100 subjects from the test-set were randomly selected for manual measurement of lumbar lordosis (LL), sacral slope (SS), pelvic incidence (PI), and sagittal vertical axis (SVA) by two radiologists in both synthetic and real images.

RESULTS

Sixty-nine synthetic images were considered assessable. The intraclass correlation coefficient ranged 0.93-0.99 for measurements in real images, and from 0.83 to 0.88 for synthetic images. Correlations between parameters of real and synthetic images were 0.52 (LL), 0.17 (SS), 0.18 (PI), and 0.74 (SVA). Measurement errors showed minimal correlation with scoliosis severity. Mean ± standard deviation absolute errors were 7 ± 7° (LL), 9 ± 7° (SS), 9 ± 8° (PI), and 1.1 ± 0.8 cm (SVA).

CONCLUSION

While the model generates sagittal images visually consistent with reference images, their quality is not sufficient for clinical parameter assessment, except for promising results in SVA.

RELEVANCE STATEMENT

AI can generate synthetic sagittal radiographs from coronal views to reduce radiation exposure in monitoring adolescent idiopathic scoliosis (AIS). However, while these synthetic images appear visually consistent with real ones, their quality remains insufficient for accurate clinical assessment.

KEY POINTS

AI can be exploited to generate synthetic sagittal radiographs from coronal views. Dataset of 3,935 subjects was used to train and test AI-model; spinal parameters from synthetic and real images were compared. Synthetic images were visually consistent with real ones, but quality was generally insufficient for accurate clinical assessment.

Generative Artificial Intelligence in Pathology and Medicine: A Deeper Dive

This review article builds upon the introductory piece in our 7-part series, delving deeper into the transformative potential of generative artificial intelligence (Gen AI) in pathology and medicine. The article explores the applications of Gen AI models in pathology and medicine, including the use of custom chatbots for diagnostic report generation, synthetic image synthesis for training new models, data set augmentation, hypothetical scenario generation for educational purposes, and the use of multimodal along with multiagent models. This article also provides an overview of the common categories within Gen AI models, discussing open-source and closed-source models, as well as specific examples of popular models such as GPT-4, Llama, Mistral, DALL-E, Stable Diffusion, and their associated frameworks (eg, transformers, generative adversarial networks, diffusion-based neural networks), along with their limitations and challenges, especially within the medical domain. We also review common libraries and tools that are currently deemed necessary to build and integrate such models. Finally, we look to the future, discussing the potential impact of Gen AI on health care, including benefits, challenges, and concerns related to privacy, bias, ethics, application programming interface costs, and security measures.

Unsupervised Bayesian generation of synthetic CT from CBCT using patient-specific score-based prior

BACKGROUND

Cone-beam computed tomography (CBCT) scans, performed fractionally (e.g., daily or weekly), are widely utilized for patient alignment in the image-guided radiotherapy (IGRT) process, thereby making it a potential imaging modality for the implementation of adaptive radiotherapy (ART) protocols. Nonetheless, significant artifacts and incorrect Hounsfield unit (HU) values hinder their application in quantitative tasks such as target and organ segmentations and dose calculation. Therefore, acquiring CT-quality images from the CBCT scans is essential to implement online ART in clinical settings.

PURPOSE

This work aims to develop an unsupervised learning method using the patient-specific diffusion model for CBCT-based synthetic CT (sCT) generation to improve the image quality of CBCT.

METHODS

The proposed method is in an unsupervised framework that utilizes a patient-specific score-based model as the image prior alongside a customized total variation (TV) regularization to enforce coherence across different transverse slices. The score-based model is unconditionally trained using the same patient's planning CT (pCT) images to characterize the manifold of CT-quality images and capture the unique anatomical information of the specific patient. The efficacy of the proposed method was assessed on images from anatomical sites including head and neck (H&N) cancer, pancreatic cancer, and lung cancer. The performance of the proposed CBCT correction method was evaluated using quantitative metrics, including mean absolute error (MAE), non-uniformity (NU), and structural similarity index measure (SSIM). Additionally, the proposed algorithm was benchmarked against other unsupervised learning-based CBCT correction algorithms.

RESULTS

The proposed method significantly reduced various kinds of CBCT artifacts in the studies of H&N, pancreatic, and lung cancer patients. In the lung stereotactic body radiation therapy (SBRT) patient study, the MAE, NU, and SSIM were improved from 47 HU, 45 HU, and 0.58 in the original CBCT images to 13 HU, 14 dB, and 0.67 in the generated sCT images. Compared to other unsupervised learning-based algorithms, the proposed method demonstrated superior performance in artifact reduction.

CONCLUSIONS

The proposed unsupervised method can generate sCT from CBCT with reduced artifacts and precise HU values, enabling CBCT-guided segmentation and replanning for online ART.

Deep learning methods for 3D magnetic resonance image denoising, bias field and motion artifact correction: a comprehensive review

Magnetic resonance imaging (MRI) provides detailed structural information of the internal body organs and soft tissue regions of a patient in clinical diagnosis for disease detection, localization, and progress monitoring. MRI scanner hardware manufacturers incorporate various post-acquisition image-processing techniques into the scanner's computer software tools for different post-processing tasks. These tools provide a final image of adequate quality and essential features for accurate clinical reporting and predictive interpretation for better treatment planning. Different post-acquisition image-processing tasks for MRI quality enhancement include noise removal, motion artifact reduction, magnetic bias field correction, and eddy electric current effect removal. Recently, deep learning (DL) methods have shown great success in many research fields, including image and video applications. DL-based data-driven feature-learning approaches have great potential for MR image denoising and image-quality-degrading artifact correction. Recent studies have demonstrated significant improvements in image-analysis tasks using DL-based convolutional neural network techniques. The promising capabilities and performance of DL techniques in various problem-solving domains have motivated researchers to adapt DL methods to medical image analysis and quality enhancement tasks. This paper presents a comprehensive review of DL-based state-of-the-art MRI quality enhancement and artifact removal methods for regenerating high-quality images while preserving essential anatomical and physiological feature maps without destroying important image information. Existing research gaps and future directions have also been provided by highlighting potential research areas for future developments, along with their importance and advantages in medical imaging.

Advancements in synthetic CT generation from MRI: A review of techniques, and trends in radiation therapy planning

BACKGROUND

Magnetic resonance imaging (MRI) and Computed tomography (CT) are crucial imaging techniques in both diagnostic imaging and radiation therapy. MRI provides excellent soft tissue contrast but lacks the direct electron density data needed to calculate dosage. CT, on the other hand, remains the gold standard due to its accurate electron density information in radiation therapy planning (RTP) but it exposes patients to ionizing radiation. Synthetic CT (sCT) generation from MRI has been a focused study field in the last few years due to cost effectiveness as well as for the objective of minimizing side-effects of using more than one imaging modality for treatment simulation. It offers significant time and cost efficiencies, bypassing the complexities of co-registration, and potentially improving treatment accuracy by minimizing registration-related errors. In an effort to navigate the quickly developing field of precision medicine, this paper investigates recent advancements in sCT generation techniques, particularly those using machine learning (ML) and deep learning (DL). The review highlights the potential of these techniques to improve the efficiency and accuracy of sCT generation for use in RTP by improving patient care and reducing healthcare costs. The intricate web of sCT generation techniques is scrutinized critically, with clinical implications and technical underpinnings for enhanced patient care revealed.

PURPOSE

This review aims to provide an overview of the most recent advancements in sCT generation from MRI with a particular focus of its use within RTP, emphasizing on techniques, performance evaluation, clinical applications, future research trends and open challenges in the field.

METHODS

A thorough search strategy was employed to conduct a systematic literature review across major scientific databases. Focusing on the past decade's advancements, this review critically examines emerging approaches introduced from 2013 to 2023 for generating sCT from MRI, providing a comprehensive analysis of their methodologies, ultimately fostering further advancement in the field. This study highlighted significant contributions, identified challenges, and provided an overview of successes within RTP. Classifying the identified approaches, contrasting their advantages and disadvantages, and identifying broad trends were all part of the review's synthesis process.

RESULTS

The review identifies various sCT generation approaches, consisting atlas-based, segmentation-based, multi-modal fusion, hybrid approaches, ML and DL-based techniques. These approaches are evaluated for image quality, dosimetric accuracy, and clinical acceptability. They are used for MRI-only radiation treatment, adaptive radiotherapy, and MR/PET attenuation correction. The review also highlights the diversity of methodologies for sCT generation, each with its own advantages and limitations. Emerging trends incorporate the integration of advanced imaging modalities including various MRI sequences like Dixon sequences, T1-weighted (T1W), T2-weighted (T2W), as well as hybrid approaches for enhanced accuracy.

CONCLUSIONS

The study examines MRI-based sCT generation, to minimize negative effects of acquiring both modalities. The study reviews 2013-2023 studies on MRI to sCT generation methods, aiming to revolutionize RTP by reducing use of ionizing radiation and improving patient outcomes. The review provides insights for researchers and practitioners, emphasizing the need for standardized validation procedures and collaborative efforts to refine methods and address limitations. It anticipates the continued evolution of techniques to improve the precision of sCT in RTP.

Exploring dual energy CT synthesis in CBCT-based adaptive radiotherapy and proton therapy: application of denoising diffusion probabilistic models

Background.Adaptive radiotherapy (ART) requires precise tissue characterization to optimize treatment plans and enhance the efficacy of radiation delivery while minimizing exposure to organs at risk. Traditional imaging techniques such as cone beam computed tomography (CBCT) used in ART settings often lack the resolution and detail necessary for accurate dosimetry, especially in proton therapy.Purpose.This study aims to enhance ART by introducing an innovative approach that synthesizes dual-energy computed tomography (DECT) images from CBCT scans using a novel 3D conditional denoising diffusion probabilistic model (DDPM) multi-decoder. This method seeks to improve dose calculations in ART planning, enhancing tissue characterization.Methods.We utilized a paired CBCT-DECT dataset from 54 head and neck cancer patients to train and validate our DDPM model. The model employs a multi-decoder Swin-UNET architecture that synthesizes high-resolution DECT images by progressively reducing noise and artifacts in CBCT scans through a controlled diffusion process.Results.The proposed method demonstrated superior performance in synthesizing DECT images (High DECT MAE 39.582 ± 0.855 and Low DECT MAE 48.540± 1.833) with significantly enhanced signal-to-noise ratio and reduced artifacts compared to traditional GAN-based methods. It showed marked improvements in tissue characterization and anatomical structure similarity, critical for precise proton and radiation therapy planning.Conclusions.This research has opened a new avenue in CBCT-CT synthesis for ART/APT by generating DECT images using an enhanced DDPM approach. The demonstrated similarity between the synthesized DECT images and ground truth images suggests that these synthetic volumes can be used for accurate dose calculations, leading to better adaptation in treatment planning.

Results of 2023 survey on the use of synthetic computed tomography for magnetic resonance Imaging-only radiotherapy: Current status and future steps

Background and purpose

The emergence of synthetic CT (sCT) in MR-guided radiotherapy (MRgRT) represents a significant advancement, supporting MR-only workflows and online treatment adaptation. However, the lack of consensus guidelines has led to varied practices. This study reports results from a 2023 ESTRO survey aimed at defining current practices in sCT development and use.

Materials and methods

An survey was distributed to ESTRO members, including 98 questions across four sections on sCT algorithm generation and usage. By June 2023, 100 centers participated. The survey revealed diverse clinical experiences and roles, with primary sCT use in the pelvis (60%), brain (15%), abdomen (11%), thorax (8%), and head-and-neck (6%). sCT was mostly used for conventional fractionation treatments (68%), photon SBRT (40%), and palliative cases (28%), with limited use in proton therapy (4%).

Results

Conditional GANs and GANs were the most used neural network architectures, operating mainly on 1.5 T and 3 T MRI images. Less than half used paired images for training, and only 20% performed image selection. Key MR image quality parameters included magnetic field homogeneity and spatial integrity. Half of the respondents lacked a dedicated sCT-QA program, and many did not apply sanitychecks before calculation. Selection strategies included age, weight, and metal artifacts. A strong consensus (95%) emerged for vendor neutral guidelines.

Conclusion

The survey highlights the need for expert-based, vendor-neutral guidelines to standardize sCT tools, metrics, and clinical protocols, ensuring effective sCT use in MR-guided radiotherapy.

Challenges and opportunities in the development and clinical implementation of artificial intelligence based synthetic computed tomography for magnetic resonance only radiotherapy

Synthetic computed tomography (sCT) generated from magnetic resonance imaging (MRI) can serve as a substitute for planning CT in radiation therapy (RT), thereby removing registration uncertainties associated with multi-modality imaging pairing, reducing costs and patient radiation exposure. CE/FDA-approved sCT solutions are nowadays available for pelvis, brain, and head and neck, while more complex deep learning (DL) algorithms are under investigation for other anatomic sites. The main challenge in achieving a widespread clinical implementation of sCT lies in the absence of consensus on sCT commissioning and quality assurance (QA), resulting in variation of sCT approaches across different hospitals. To address this issue, a group of experts gathered at the ESTRO Physics Workshop 2022 to discuss the integration of sCT solutions into clinics and report the process and its outcomes. This position paper focuses on aspects of sCT development and commissioning, outlining key elements crucial for the safe implementation of an MRI-only RT workflow.

A Deep Learning Approach for the Fast Generation of Synthetic Computed Tomography from Low-Dose Cone Beam Computed Tomography Images on a Linear Accelerator Equipped with Artificial Intelligence

Artificial Intelligence (AI) is revolutionising many aspects of radiotherapy (RT), opening scenarios that were unimaginable just a few years ago. The aim of this study is to propose a Deep Leaning (DL) approach able to quickly generate synthetic Computed Tomography (CT) images from low-dose Cone Beam CT (CBCT) acquired on a modern linear accelerator integrating AI. Methods: A total of 53 patients treated in the pelvic region were enrolled and split into training (30), validation (9), and testing (14). A Generative Adversarial Network (GAN) was trained for 200 epochs. The image accuracy was evaluated by calculating the mean and mean absolute error (ME and ME) between sCT and CT. RT treatment plans were calculated on CT and sCT images, and dose accuracy was evaluated considering Dose Volume Histogram (DVH) and gamma analysis. Results: A total of 4507 images were selected for training. The MAE and ME values in the test set were 36 ± 6 HU and 7 ± 6 HU, respectively. Mean gamma passing rates for 1%/1 mm, 2%/2 mm, and 3%/3 mm tolerance criteria were respectively 93.5 ± 3.4%, 98.0 ± 1.3%, and 99.2 ± 0.7%, with no difference between curative and palliative cases. All the DVH parameters analysed were within 1 Gy of the difference between sCT and CT. Conclusion: This study demonstrated that sCT generation using the DL approach is feasible on low-dose CBCT images. The proposed approach can represent a valid tool to speed up the online adaptive procedure and remove CT simulation from the RT workflow.

Fast MRI Reconstruction Using Deep Learning-based Compressed Sensing: A Systematic Review

Magnetic resonance imaging (MRI) has revolutionized medical imaging, providing a non-invasive and highly detailed look into the human body. However, the long acquisition times of MRI present challenges, causing patient discomfort, motion artifacts, and limiting real-time applications. To address these challenges, researchers are exploring various techniques to reduce acquisition time and improve the overall efficiency of MRI. One such technique is compressed sensing (CS), which reduces data acquisition by leveraging image sparsity in transformed spaces. In recent years, deep learning (DL) has been integrated with CS-MRI, leading to a new framework that has seen remarkable growth. DL-based CS-MRI approaches are proving to be highly effective in accelerating MR imaging without compromising image quality. This review comprehensively examines DL-based CS-MRI techniques, focusing on their role in increasing MR imaging speed. We provide a detailed analysis of each category of DL-based CS-MRI including end-to-end, unroll optimization, self-supervised, and federated learning. Our systematic review highlights significant contributions and underscores the exciting potential of DL in CS-MRI. Additionally, our systematic review efficiently summarizes key results and trends in DL-based CS-MRI including quantitative metrics, the dataset used, acceleration factors, and the progress of and research interest in DL techniques over time. Finally, we discuss potential future directions and the importance of DL-based CS-MRI in the advancement of medical imaging. To facilitate further research in this area, we provide a GitHub repository that includes up-to-date DL-based CS-MRI publications and publicly available datasets - https://github.com/mosaf/Awesome-DL-based-CS-MRI.

Synthetic CT imaging for PET monitoring in proton therapy: a simulation study

Objective.This study addresses a fundamental limitation of in-beam positron emission tomography (IB-PET) in proton therapy: the lack of direct anatomical representation in the images it produces. We aim to overcome this shortcoming by pioneering the application of deep learning techniques to create synthetic control CT images (sCT) from combining IB-PET and planning CT scan data.Approach.We conducted simulations involving six patients who underwent irradiation with proton beams. Leveraging the architecture of a visual transformer (ViT) neural network, we developed a model to generate sCT images of these patients using the planning CT scans and the inter-fractional simulated PET activity maps during irradiation. To evaluate the model's performance, a comparison was conducted between the sCT images produced by the ViT model and the authentic control CT images-serving as the benchmark.Main results.The structural similarity index was computed at a mean value across all patients of 0.91, while the mean absolute error measured 22 Hounsfield Units (HU). Root mean squared error and peak signal-to-noise ratio values were 56 HU and 30 dB, respectively. The Dice similarity coefficient exhibited a value of 0.98. These values are comparable to or exceed those found in the literature. More than 70% of the synthetic morphological changes were found to be geometrically compatible with the ones reported in the real control CT scan.Significance.Our study presents an innovative approach to surface the hidden anatomical information of IB-PET in proton therapy. Our ViT-based model successfully generates sCT images from inter-fractional PET data and planning CT scans. Our model's performance stands on par with existing models relying on input from cone beam CT or magnetic resonance imaging, which contain more anatomical information than activity maps.

Deep conditional generative model for longitudinal single-slice abdominal computed tomography harmonization

Purpose

Two-dimensional single-slice abdominal computed tomography (CT) provides a detailed tissue map with high resolution allowing quantitative characterization of relationships between health conditions and aging. However, longitudinal analysis of body composition changes using these scans is difficult due to positional variation between slices acquired in different years, which leads to different organs/tissues being captured.

Approach

To address this issue, we propose C-SliceGen, which takes an arbitrary axial slice in the abdominal region as a condition and generates a pre-defined vertebral level slice by estimating structural changes in the latent space.

Results

Our experiments on 2608 volumetric CT data from two in-house datasets and 50 subjects from the 2015 Multi-Atlas Abdomen Labeling Challenge Beyond the Cranial Vault (BTCV) dataset demonstrate that our model can generate high-quality images that are realistic and similar. We further evaluate our method's capability to harmonize longitudinal positional variation on 1033 subjects from the Baltimore longitudinal study of aging dataset, which contains longitudinal single abdominal slices, and confirmed that our method can harmonize the slice positional variance in terms of visceral fat area.

Conclusion

This approach provides a promising direction for mapping slices from different vertebral levels to a target slice and reducing positional variance for single-slice longitudinal analysis. The source code is available at: https://github.com/MASILab/C-SliceGen.