Iterative framework for the joint segmentation and CT synthesis of MR images: application to MRI-only radiotherapy treatment planning

One-shot neuroanatomy segmentation through online data augmentation and confidence aware pseudo label

Recently, deep learning-based brain segmentation methods have achieved great success. However, most approaches focus on supervised segmentation, which requires many high-quality labeled images. In this paper, we pay attention to one-shot segmentation, aiming to learn from one labeled image and a few unlabeled images. We propose an end-to-end unified network that joints deformation modeling and segmentation tasks. Our network consists of a shared encoder, a deformation modeling head, and a segmentation head. In the training phase, the atlas and unlabeled images are input to the encoder to get multi-scale features. The features are then fed to the multi-scale deformation modeling module to estimate the atlas-to-image deformation field. The deformation modeling module implements the estimation at the feature level in a coarse-to-fine manner. Then, we employ the field to generate the augmented image pair through online data augmentation. We do not apply any appearance transformations cause the shared encoder could capture appearance variations. Finally, we adopt supervised segmentation loss for the augmented image. Considering that the unlabeled images still contain rich information, we introduce confidence aware pseudo label for them to further boost the segmentation performance. We validate our network on three benchmark datasets. Experimental results demonstrate that our network significantly outperforms other deep single-atlas-based and traditional multi-atlas-based segmentation methods. Notably, the second dataset is collected from multi-center, and our network still achieves promising segmentation performance on both the seen and unseen test sets, revealing its robustness. The source code will be available at https://github.com/zhangliutong/brainseg.

Uncertainty-aware MR-based CT synthesis for robust proton therapy planning of brain tumour

BACKGROUND AND PURPOSE

Deep learning techniques excel in MR-based CT synthesis, but missing uncertainty prediction limits its clinical use in proton therapy. We developed an uncertainty-aware framework and evaluated its efficiency in robust proton planning.

MATERIALS AND METHODS

A conditional generative-adversarial network was trained on 64 brain tumour patients with paired MR-CT images to generate synthetic CTs (sCT) from combined T1-T2 MRs of three orthogonal planes. A Bayesian neural network predicts Laplacian distributions for all voxels with parameters (μ, b). A robust proton plan was optimized using three sCTs of μ and μ±b. The dosimetric differences between the plan from sCT (sPlan) and the recalculated plan (rPlan) on planning CT (pCT) were quantified for each patient. The uncertainty-aware robust plan was compared to conventional robust (global ± 3 %) and non-robust plans.

RESULTS

In 8-fold cross-validation, sCT-pCT image differences (Mean-Absolute-Error) were 80.84 ± 9.84HU (body), 35.78 ± 6.07HU (soft tissues) and 221.88 ± 31.69HU (bones), with Dice scores of 90.33 ± 2.43 %, 95.13 ± 0.80 %, and 85.53 ± 4.16 %, respectively. The uncertainty distribution positively correlated with absolute prediction error (Correlation Coefficient: 0.62 ± 0.01). The uncertainty-conditioned robust optimisation improved the rPlan-sPlan agreement, e.g., D95 absolute difference (CTV) was 1.10 ± 1.24 % compared to conventional (1.64 ± 2.71 %) and non-robust (2.08 ± 2.96 %) optimisation. This trend was consistent across all target and organs-at-risk indexes.

CONCLUSION

The enhanced framework incorporates 3D uncertainty prediction and generates high-quality sCTs from MR images. The framework also facilitates conditioned robust optimisation, bolstering proton plan robustness against network prediction errors. The innovative feature of uncertainty visualisation and robust analyses contribute to evaluating sCT clinical utility for individual patients.

Deep-Learning-Aided Evaluation of Spondylolysis Imaged with Ultrashort Echo Time Magnetic Resonance Imaging

Isthmic spondylolysis results in fracture of pars interarticularis of the lumbar spine, found in as many as half of adolescent athletes with persistent low back pain. While computed tomography (CT) is the gold standard for the diagnosis of spondylolysis, the use of ionizing radiation near reproductive organs in young subjects is undesirable. While magnetic resonance imaging (MRI) is preferable, it has lowered sensitivity for detecting the condition. Recently, it has been shown that ultrashort echo time (UTE) MRI can provide markedly improved bone contrast compared to conventional MRI. To take UTE MRI further, we developed supervised deep learning tools to generate (1) CT-like images and (2) saliency maps of fracture probability from UTE MRI, using ex vivo preparation of cadaveric spines. We further compared quantitative metrics of the contrast-to-noise ratio (CNR), mean squared error (MSE), peak signal-to-noise ratio (PSNR), and structural similarity index (SSIM) between UTE MRI (inverted to make the appearance similar to CT) and CT and between CT-like images and CT. Qualitative results demonstrated the feasibility of successfully generating CT-like images from UTE MRI to provide easier interpretability for bone fractures thanks to improved image contrast and CNR. Quantitatively, the mean CNR of bone against defect-filled tissue was 35, 97, and 146 for UTE MRI, CT-like, and CT images, respectively, being significantly higher for CT-like than UTE MRI images. For the image similarity metrics using the CT image as the reference, CT-like images provided a significantly lower mean MSE (0.038 vs. 0.0528), higher mean PSNR (28.6 vs. 16.5), and higher SSIM (0.73 vs. 0.68) compared to UTE MRI images. Additionally, the saliency maps enabled quick detection of the location with probable pars fracture by providing visual cues to the reader. This proof-of-concept study is limited to the data from ex vivo samples, and additional work in human subjects with spondylolysis would be necessary to refine the models for clinical use. Nonetheless, this study shows that the utilization of UTE MRI and deep learning tools could be highly useful for the evaluation of isthmic spondylolysis.

MR-CT multi-atlas registration guided by fully automated brain structure segmentation with CNNs

PURPOSE

Computed tomography (CT) is widely used to identify anomalies in brain tissues because their localization is important for diagnosis and therapy planning. Due to the insufficient soft tissue contrast of CT, the division of the brain into anatomical meaningful regions is challenging and is commonly done with magnetic resonance imaging (MRI).

METHODS

We propose a multi-atlas registration approach to propagate anatomical information from a standard MRI brain atlas to CT scans. This translation will enable a detailed automated reporting of brain CT exams. We utilize masks of the lateral ventricles and the brain volume of CT images as adjuvant input to guide the registration process. Besides using manual annotations to test the registration in a first step, we then verify that convolutional neural networks (CNNs) are a reliable solution for automatically segmenting structures to enhance the registration process.

RESULTS

The registration method obtains mean Dice values of 0.92 and 0.99 in brain ventricles and parenchyma on 22 healthy test cases when using manually segmented structures as guidance. When guiding with automatically segmented structures, the mean Dice values are 0.87 and 0.98, respectively.

CONCLUSION

Our registration approach is a fully automated solution to register MRI atlas images to CT scans and thus obtain detailed anatomical information. The proposed CNN segmentation method can be used to obtain masks of ventricles and brain volume which guide the registration.

Pseudo computed tomography image generation from brain magnetic resonance image using integration of PCA & DCNN-UNET: A comparative analysis

MRI-Only Radiation (RT) now avoids some of the issues associated with employing Computed Tomography(CT) in RT chains, such as MRI registration to a separate CT, excess dosage administration, and the cost of recurrent imaging. The fact that MRI signal intensities are unrelated to the biological tissue’s attenuation coefficient poses a problem. This raises workloads, creates uncertainty as a result of the required inter-modality image registrations, and exposes patients to needless radiation. While using only MRI would be preferable, a method for estimating a pseudo-CT (pCT)or synthetic-CT(sCT) for producing electron density maps and patient positioning reference images is required. As Deep Learning(DL) is revolutionized in so many fields these days, an effective and accurate model is required for generating pCT from MRI. So, this paper depicts an efficient DL model in which the following are the stages: a) Data Acquisition where CT and MRI images are collected b) preprocessing these to avoid the anomalies and noises using techniques like outlier elimination, data smoothening and data normalizing c) feature extraction and selection using Principal Component Analysis (PCA) & regression method d) generating pCT from MRI using Deep Convolutional Neural Network and UNET (DCNN-UNET). We here compare both feature extraction (PCA) and classification model (DCNN-UNET) with other methods such as Discrete Wavelet Tranform(DWT), Independent Component Analysis(ICA), Fourier Transform and VGG16, ResNet, AlexNet, DenseNet, CNN (Convolutional Neural Network)respectively. The performance measures used to evaluate these models are Dice Coefficient(DC), Structured Similarity Index Measure(SSIM), Mean Absolute Error(MAE), Mean Squared Error(MSE), Accuracy, Computation Time in which our proposed system outperforms better with 0.94±0.02 over other state-of-art models.

Anomaly detection for the individual analysis of brain PET images

Purpose: In clinical practice, positron emission tomography (PET) images are mostly analyzed visually, but the sensitivity and specificity of this approach greatly depend on the observer's experience. Quantitative analysis of PET images would alleviate this problem by helping define an objective limit between normal and pathological findings. We present an anomaly detection framework for the individual analysis of PET images. Approach: We created subject-specific abnormality maps that summarize the pathology's topographical distribution in the brain by comparing the subject's PET image to a model of healthy PET appearance that is specific to the subject under investigation. This model was generated from demographically and morphologically matched PET scans from a control dataset. Results: We generated abnormality maps for healthy controls, patients at different stages of Alzheimer's disease and with different frontotemporal dementia syndromes. We showed that no anomalies were detected for the healthy controls and that the anomalies detected from the patients with dementia coincided with the regions where abnormal uptake was expected. We also validated the proposed framework using the abnormality maps as inputs of a classifier and obtained higher classification accuracies than when using the PET images themselves as inputs. Conclusions: The proposed method was able to automatically locate and characterize the areas characteristic of dementia from PET images. The abnormality maps are expected to (i) help clinicians in their diagnosis by highlighting, in a data-driven fashion, the pathological areas, and (ii) improve the interpretability of subsequent analyses, such as computer-aided diagnosis or spatiotemporal modeling.

Generation of Pseudo-CT using High-Degree Polynomial Regression on Dual-Contrast Pelvic MRI Data

Increasing interests in using magnetic resonance imaging only in radiation therapy require methods for predicting the computed tomography numbers from MRI data. Here we propose a simple voxel method to generate the pseudo-CT (pCT) image using dual-contrast pelvic MRI data. The method is first trained with the CT data and dual-contrast MRI data (two sets of MRI with different sequences) of multiple patients, where the anatomical structures in the images after deformable image registration are segmented into several regions, and after MRI intensity normalizations a regression analysis is used to determine a two-variable polynomial function for each region to relate a voxel’s two MRI intensity values to its CT number. We first evaluate the accuracy via the Hounsfield unit (HU) difference between the pseudo-CT and reference-CT (rCT) images and obtain the average mean absolute error as 40.3 ± 2.9 HU from leave-one-out-cross-validation (LOOCV) across all six patients, which is better than most previous results and comparable to another study using the more complicated atlas-based method. We also perform a dosimetric evaluation of the treatment plans based on pCT and rCT images and find the average passing rate within 2% dose difference to be 95.4% in point-to-point dose comparisons. Therefore, our method shows encouraging results in predicting the CT numbers. This polynomial method needs less computer storage than the interpolation method and can be readily extended to the case of more than two MRI sequences.

Pseudo-CT generation from multi-parametric MRI using a novel multi-channel multi-path conditional generative adversarial network for nasopharyngeal carcinoma patients

PURPOSE

To develop and evaluate a novel method for pseudo-CT generation from multi-parametric MR images using multi-channel multi-path generative adversarial network (MCMP-GAN).

METHODS

Pre- and post-contrast T1-weighted (T1-w), T2-weighted (T2-w) MRI, and treatment planning CT images of 32 nasopharyngeal carcinoma (NPC) patients were employed to train a pixel-to-pixel MCMP-GAN. The network was developed based on a 5-level Residual U-Net (ResU-Net) with the channel-based independent feature extraction network to generate pseudo-CT images from multi-parametric MR images. The discriminator with five convolutional layers was added to distinguish between the real CT and pseudo-CT images, improving the nonlinearity and prediction accuracy of the model. Eightfold cross validation was implemented to validate the proposed MCMP-GAN. The pseudo-CT images were evaluated against the corresponding planning CT images based on mean absolute error (MAE), peak signal-to-noise ratio (PSNR), Dice similarity coefficient (DSC), and Structural similarity index (SSIM). Similar comparisons were also performed against the multi-channel single-path GAN (MCSP-GAN), the single-channel single-path GAN (SCSP-GAN).

RESULTS

It took approximately 20 h to train the MCMP-GAN model on a Quadro P6000, and less than 10 s to generate all pseudo-CT images for the subjects in the test set. The average head MAE between pseudo-CT and planning CT was 75.7 ± 14.6 Hounsfield Units (HU) for MCMP-GAN, significantly (P-values < 0.05) lower than that for MCSP-GAN (79.2 ± 13.0 HU) and SCSP-GAN (85.8 ± 14.3 HU). For bone only, the MCMP-GAN yielded a smaller mean MAE (194.6 ± 38.9 HU) than MCSP-GAN (203.7 ± 33.1 HU), SCSP-GAN (227.0 ± 36.7 HU). The average PSNR of MCMP-GAN (29.1 ± 1.6) was found to be higher than that of MCSP-GAN (28.8 ± 1.2) and SCSP-GAN (28.2 ± 1.3). In terms of metrics for image similarity, MCMP-GAN achieved the highest SSIM (0.92 ± 0.02) but did not show significantly improved bone DSC results in comparison with MCSP-GAN.

CONCLUSIONS

We developed a novel multi-channel GAN approach for generating pseudo-CT from multi-parametric MR images. Our preliminary results in NPC patients showed that the MCMP-GAN method performed apparently superior to the U-Net-GAN and SCSP-GAN, and slightly better than MCSP-GAN.

Evaluation of a deep learning-based pelvic synthetic CT generation technique for MRI-based prostate proton treatment planning

The purpose of this work is to validate the application of a deep learning-based method for pelvic synthetic CT (sCT) generation that can be used for prostate proton beam therapy treatment planning. We propose to integrate dense block minimization into 3D cycle-consistent generative adversarial networks (cycleGAN) framework to effectively learn the nonlinear mapping between MRI and CT pairs. A cohort of 17 patients with co-registered CT and MR pairs were used to test the deep learning-based sCT generation method by leave-one-out cross-validation. Image quality between the sCT and CT images, gamma analysis passing rate, dose-volume metrics, distal range displacement, and the individual pencil beam Bragg peak shift between sCT- and CT-based proton plans were evaluated. The average mean absolute error (MAE) was 51.32  ±  16.91 HU. The relative differences of the statistics of the PTV dose-volume histogram (DVH) metrics in between sCT and CT were generally less than 1%. Mean values of dose difference, absolute dose difference (in percent of the prescribed dose) were  -0.07%  ±  0.07% and 0.23%  ±  0.08%. Mean gamma analysis pass rate of 1 mm/1%, 2 mm/2%, 3 mm/3% criteria with 10% dose threshold were 92.39%  ±  5.97%, 97.95%  ±  2.95% and 98.97%  ±  1.62% respectively. The median, mean and standard deviation of absolute maximum range differences were 0.09 cm and 0.23  ±  0.25 cm. The median and mean Bragg peak shifts among the 17 patients were 0.09 cm and 0.18  ±  0.07 cm. The image similarity, dosimetric and distal range agreement between sCT and original CT suggests the feasibility of further development of an MRI-only workflow for prostate proton radiotherapy.

Flexible Prediction of CT Images From MRI Data Through Improved Neighborhood Anchored Regression for PET Attenuation Correction

Given the complicated relationship between the magnetic resonance imaging (MRI) signals and the attenuation values, the attenuation correction in hybrid positron emission tomography (PET)/MRI systems remains a challenging task. Currently, existing methods are either time-consuming or require sufficient samples to train the models. In this paper, an efficient approach for predicting pseudo computed tomography (CT) images from T1- and T2-weighted MRI data with limited data is proposed. The proposed approach uses improved neighborhood anchored regression (INAR) as a baseline method to pre-calculate projected matrices to flexibly predict the pseudo CT patches. Techniques, including the augmentation of the MR/CT dataset, learning of the nonlinear descriptors of MR images, hierarchical search for nearest neighbors, data-driven optimization, and multi-regressor ensemble, are adopted to improve the effectiveness of the proposed approach. In total, 22 healthy subjects were enrolled in the study. The pseudo CT images obtained using INAR with multi-regressor ensemble yielded mean absolute error (MAE) of 92.73 ± 14.86 HU, peak signal-to-noise ratio of 29.77 ± 1.63 dB, Pearson linear correlation coefficient of 0.82 ± 0.05, dice similarity coefficient of 0.81 ± 0.03, and the relative mean absolute error (rMAE) in PET attenuation correction of 1.30 ± 0.20% compared with true CT images. Moreover, our proposed INAR method, without any refinement strategies, can achieve considerable results with only seven subjects (MAE 106.89 ± 14.43 HU, rMAE 1.51 ± 0.21%). The experiments prove the superior performance of the proposed method over the six innovative methods. Moreover, the proposed method can rapidly generate the pseudo CT images that are suitable for PET attenuation correction.

Automatic Labeling of MR Brain Images Through the Hashing Retrieval Based Atlas Forest

The multi-atlas method is one of the efficient and common automatic labeling method, which uses the prior information provided by expert-labeled images to guide the labeling of the target. However, most multi-atlas-based methods depend on the registration that may not give the correct information during the label propagation. To address the issue, we designed a new automatic labeling method through the hashing retrieval based atlas forest. The proposed method propagates labels without registration to reduce the errors, and constructs a target-oriented learning model to integrate information among the atlases. This method innovates a coarse classification strategy to preprocess the dataset, which retains the integrity of dataset and reduces computing time. Furthermore, the method considers each voxel in the atlas as a sample and encodes these samples with hashing for the fast sample retrieval. In the stage of labeling, the method selects suitable samples through hashing learning and trains atlas forests by integrating the information from the dataset. Then, the trained model is used to predict the labels of the target. Experimental results on two datasets illustrated that the proposed method is promising in the automatic labeling of MR brain images.

Association of bone mineral density and fat fraction with magnetic susceptibility in inflamed trabecular bone

PURPOSE

To evaluate the relationship between bone mineral density (BMD) and magnetic susceptibility, and between proton density fat fraction and susceptibility, in inflamed trabecular bone.

METHODS

Two different phantoms modeling the fat fraction (FF) and BMD values of healthy bone marrow and disease states were scanned using a multiecho gradient echo acquisition at 3T. After correction for fat-water chemical shift, susceptibility mapping was performed, and susceptibility measurements were compared with BMD and FF values using linear regression. Patients with spondyloarthritis were scanned using the same protocol, and susceptibility values were calculated in areas of inflamed bone (edema) and fat metaplasia, both before and after accounting for the contribution of fat to the total susceptibility.

RESULTS

Susceptibility values in the phantoms were accurately described by a 2D linear function, with a negative correlation between BMD and susceptibility and a positive correlation between FF and susceptibility (adjusted R2 = 0.77; P = 3·10-5 ). In patients, significant differences in susceptibility were observed between fat metaplasia and normal marrow, but these differences were eliminated by removing the fat contribution to the total susceptibility.

CONCLUSIONS

BMD and proton density fat fraction both influence the total susceptibility of bone marrow and failure to account for the fat contribution could lead to errors in BMD quantification. We propose a method for removing the fat contribution from the total susceptibility, based on the observed linear relationship between susceptibility and FF. In inflamed bone, the overall increase in susceptibility in areas of fat metaplasia is at least partly due to increased fat content.

Comparative study of algorithms for synthetic CT generation from MRI: Consequences for MRI-guided radiation planning in the pelvic region

PURPOSE

Magnetic resonance imaging (MRI)-guided radiation therapy (RT) treatment planning is limited by the fact that the electron density distribution required for dose calculation is not readily provided by MR imaging. We compare a selection of novel synthetic CT generation algorithms recently reported in the literature, including segmentation-based, atlas-based and machine learning techniques, using the same cohort of patients and quantitative evaluation metrics.

METHODS

Six MRI-guided synthetic CT generation algorithms were evaluated: one segmentation technique into a single tissue class (water-only), four atlas-based techniques, namely, median value of atlas images (ALMedian), atlas-based local weighted voting (ALWV), bone enhanced atlas-based local weighted voting (ALWV-Bone), iterative atlas-based local weighted voting (ALWV-Iter), and a machine learning technique using deep convolution neural network (DCNN).

RESULTS

Organ auto-contouring from MR images was evaluated for bladder, rectum, bones, and body boundary. Overall, DCNN exhibited higher segmentation accuracy resulting in Dice indices (DSC) of 0.93 ± 0.17, 0.90 ± 0.04, and 0.93 ± 0.02 for bladder, rectum, and bones, respectively. On the other hand, ALMedian showed the lowest accuracy with DSC of 0.82 ± 0.20, 0.81 ± 0.08, and 0.88 ± 0.04, respectively. DCNN reached the best performance in terms of accurate derivation of synthetic CT values within each organ, with a mean absolute error within the body contour of 32.7 ± 7.9 HU, followed by the advanced atlas-based methods (ALWV: 40.5 ± 8.2 HU, ALWV-Iter: 42.4 ± 8.1 HU, ALWV-Bone: 44.0 ± 8.9 HU). ALMedian led to the highest error (52.1 ± 11.1 HU). Considering the dosimetric evaluation results, ALWV-Iter, ALWV, DCNN and ALWV-Bone led to similar mean dose estimation within each organ at risk and target volume with less than 1% dose discrepancy. However, the two-dimensional gamma analysis demonstrated higher pass rates for ALWV-Bone, DCNN, ALMedian and ALWV-Iter at 1%/1 mm criterion with 94.99 ± 5.15%, 94.59 ± 5.65%, 93.68 ± 5.53% and 93.10 ± 5.99% success, respectively, while ALWV and water-only resulted in 86.91 ± 13.50% and 80.77 ± 12.10%, respectively.

CONCLUSIONS

Overall, machine learning and advanced atlas-based methods exhibited promising performance by achieving reliable organ segmentation and synthetic CT generation. DCNN appears to have slightly better performance by achieving accurate automated organ segmentation and relatively small dosimetric errors (followed closely by advanced atlas-based methods, which in some cases achieved similar performance). However, the DCNN approach showed higher vulnerability to anatomical variation, where a greater number of outliers was observed with this method. Considering the dosimetric results obtained from the evaluated methods, the challenge of electron density estimation from MR images can be resolved with a clinically tolerable error.

Supervoxel based method for multi-atlas segmentation of brain MR images

Multi-atlas segmentation has been widely applied to the analysis of brain MR images. However, the state-of-the-art techniques in multi-atlas segmentation, including both patch-based and learning-based methods, are strongly dependent on the pairwise registration or exhibit huge spatial inconsistency. The paper proposes a new segmentation framework based on supervoxels to solve the existing challenges of previous methods. The supervoxel is an aggregation of voxels with similar attributes, which can be used to replace the voxel grid. By formulating the segmentation as a tissue labeling problem associated with a maximum-a-posteriori inference in Markov random field, the problem is solved via a graphical model with supervoxels being considered as the nodes. In addition, a dense labeling scheme is developed to refine the supervoxel labeling results, and the spatial consistency is incorporated in the proposed method. The proposed approach is robust to the pairwise registration errors and of high computational efficiency. Extensive experimental evaluations on three publically available brain MR datasets demonstrate the effectiveness and superior performance of the proposed approach.

Magnetic Resonance Imaging-Guided Adaptive Radiation Therapy: A "Game Changer" for Prostate Treatment?

Radiation therapy to the prostate involves increasingly sophisticated delivery techniques and changing fractionation schedules. With a low estimated α/β ratio, a larger dose per fraction would be beneficial, with moderate fractionation schedules rapidly becoming a standard of care. The integration of a magnetic resonance imaging (MRI) scanner and linear accelerator allows for accurate soft tissue tracking with the capacity to replan for the anatomy of the day. Extreme hypofractionation schedules become a possibility using the potentially automated steps of autosegmentation, MRI-only workflow, and real-time adaptive planning. The present report reviews the steps involved in hypofractionated adaptive MRI-guided prostate radiation therapy and addresses the challenges for implementation.