On the direct acquisition of beam's-eye-view images in MRI for integration with external beam radiotherapy

Experimental comparison of linear regression and LSTM motion prediction models for MLC-tracking on an MRI-linac

BACKGROUND

Magnetic resonance imaging (MRI)-guided radiotherapy with multileaf collimator (MLC)-tracking is a promising technique for intra-fractional motion management, achieving high dose conformality without prolonging treatment times. To improve beam-target alignment, the geometric error due to system latency should be reduced by using temporal prediction.

PURPOSE

To experimentally compare linear regression (LR) and long-short-term memory (LSTM) motion prediction models for MLC-tracking on an MRI-linac using multiple patient-derived traces with different complexities.

METHODS

Experiments were performed on a prototype 1.0 T MRI-linac capable of MLC-tracking. A motion phantom was programmed to move a target in superior-inferior (SI) direction according to eight lung cancer patient respiratory motion traces. Target centroid positions were localized from sagittal 2D cine MRIs acquired at 4 Hz using a template matching algorithm. The centroid positions were input to one of four motion prediction models. We used (1) a LSTM network which had been optimized in a previous study on patient data from another cohort (offline LSTM). We also used (2) the same LSTM model as a starting point for continuous re-optimization of its weights during the experiment based on recent motion (offline+online LSTM). Furthermore, we implemented (3) a continuously updated LR model, which was solely based on recent motion (online LR). Finally, we used (4) the last available target centroid without any changes as a baseline (no-predictor). The predictions of the models were used to shift the MLC aperture in real-time. An electronic portal imaging device (EPID) was used to visualize the target and MLC aperture during the experiments. Based on the EPID frames, the root-mean-square error (RMSE) between the target and the MLC aperture positions was used to assess the performance of the different motion predictors. Each combination of motion trace and prediction model was repeated twice to test stability, for a total of 64 experiments.

RESULTS

The end-to-end latency of the system was measured to be (389 ± 15) ms and was successfully mitigated by both LR and LSTM models. The offline+online LSTM was found to outperform the other models for all investigated motion traces. It obtained a median RMSE over all traces of (2.8 ± 1.3) mm, compared to the (3.2 ± 1.9) mm of the offline LSTM, the (3.3 ± 1.4) mm of the online LR and the (4.4 ± 2.4) mm when using the no-predictor. According to statistical tests, differences were significant (p-value <0.05) among all models in a pair-wise comparison, but for the offline LSTM and online LR pair. The offline+online LSTM was found to be more reproducible than the offline LSTM and the online LR with a maximum deviation in RMSE between two measurements of 10%.

CONCLUSIONS

This study represents the first experimental comparison of different prediction models for MRI-guided MLC-tracking using several patient-derived respiratory motion traces. We have shown that among the investigated models, continuously re-optimized LSTM networks are the most promising to account for the end-to-end system latency in MRI-guided radiotherapy with MLC-tracking.

Evaluation of real-time tumor contour prediction using LSTM networks for MR-guided radiotherapy

BACKGROUND AND PURPOSE

Magnetic resonance imaging guided radiotherapy (MRgRT) with deformable multileaf collimator (MLC) tracking would allow to tackle both rigid displacement and tumor deformation without prolonging treatment. However, the system latency must be accounted for by predicting future tumor contours in real-time. We compared the performance of three artificial intelligence (AI) algorithms based on long short-term memory (LSTM) modules for the prediction of 2D-contours 500ms into the future.

MATERIALS AND METHODS

Models were trained (52 patients, 3.1h of motion), validated (18 patients, 0.6h) and tested (18 patients, 1.1h) with cine MRs from patients treated at one institution. Additionally, we used three patients (2.9h) treated at another institution as second testing set. We implemented 1) a classical LSTM network (LSTM-shift) predicting tumor centroid positions in superior-inferior and anterior-posterior direction which are used to shift the last observed tumor contour. The LSTM-shift model was optimized both in an offline and online fashion. We also implemented 2) a convolutional LSTM model (ConvLSTM) to directly predict future tumor contours and 3) a convolutional LSTM combined with spatial transformer layers (ConvLSTM-STL) to predict displacement fields used to warp the last tumor contour.

RESULTS

The online LSTM-shift model was found to perform slightly better than the offline LSTM-shift and significantly better than the ConvLSTM and ConvLSTM-STL. It achieved a 50% Hausdorff distance of 1.2mm and 1.0mm for the two testing sets, respectively. Larger motion ranges were found to lead to more substantial performance differences across the models.

CONCLUSION

LSTM networks predicting future centroids and shifting the last tumor contour are the most suitable for tumor contour prediction. The obtained accuracy would allow to reduce residual tracking errors during MRgRT with deformable MLC-tracking.

Known Operator Learning Enables Constrained Projection Geometry Conversion: Parallel to Cone-Beam for Hybrid MR/X-Ray Imaging

X-ray imaging is a wide-spread real-time imaging technique. Magnetic Resonance Imaging (MRI) offers a multitude of contrasts that offer improved guidance to interventionalists. As such simultaneous real-time acquisition and overlay would be highly favorable for image-guided interventions, e.g., in stroke therapy. One major obstacle in this setting is the fundamentally different acquisition geometry. MRI k -space sampling is associated with parallel projection geometry, while the X-ray acquisition results in perspective distorted projections. The classical rebinning methods to overcome this limitation inherently suffers from a loss of resolution. To counter this problem, we present a novel rebinning algorithm for parallel to cone-beam conversion. We derive a rebinning formula that is then used to find an appropriate deep neural network architecture. Following the known operator learning paradigm, the novel algorithm is mapped to a neural network with differentiable projection operators enabling data-driven learning of the remaining unknown operators. The evaluation aims in two directions: First, we give a profound analysis of the different hypotheses to the unknown operator and investigate the influence of numerical training data. Second, we evaluate the performance of the proposed method against the classical rebinning approach. We demonstrate that the derived network achieves better results than the baseline method and that such operators can be trained with simulated data without losing their generality making them applicable to real data without the need for retraining or transfer learning.

Medical physics challenges in clinical MR-guided radiotherapy

The integration of magnetic resonance imaging (MRI) for guidance in external beam radiotherapy has faced significant research and development efforts in recent years. The current availability of linear accelerators with an embedded MRI unit, providing volumetric imaging at excellent soft tissue contrast, is expected to provide novel possibilities in the implementation of image-guided adaptive radiotherapy (IGART) protocols. This study reviews open medical physics issues in MR-guided radiotherapy (MRgRT) implementation, with a focus on current approaches and on the potential for innovation in IGART.Daily imaging in MRgRT provides the ability to visualize the static anatomy, to capture internal tumor motion and to extract quantitative image features for treatment verification and monitoring. Those capabilities enable the use of treatment adaptation, with potential benefits in terms of personalized medicine. The use of online MRI requires dedicated efforts to perform accurate dose measurements and calculations, due to the presence of magnetic fields. Likewise, MRgRT requires dedicated quality assurance (QA) protocols for safe clinical implementation.Reaction to anatomical changes in MRgRT, as visualized on daily images, demands for treatment adaptation concepts, with stringent requirements in terms of fast and accurate validation before the treatment fraction can be delivered. This entails specific challenges in terms of treatment workflow optimization, QA, and verification of the expected delivered dose while the patient is in treatment position. Those challenges require specialized medical physics developments towards the aim of fully exploiting MRI capabilities. Conversely, the use of MRgRT allows for higher confidence in tumor targeting and organs-at-risk (OAR) sparing.The systematic use of MRgRT brings the possibility of leveraging IGART methods for the optimization of tumor targeting and quantitative treatment verification. Although several challenges exist, the intrinsic benefits of MRgRT will provide a deeper understanding of dose delivery effects on an individual basis, with the potential for further treatment personalization.

Projection-to-Projection Translation for Hybrid X-ray and Magnetic Resonance Imaging

Hybrid X-ray and magnetic resonance (MR) imaging promises large potential in interventional medical imaging applications due to the broad variety of contrast of MRI combined with fast imaging of X-ray-based modalities. To fully utilize the potential of the vast amount of existing image enhancement techniques, the corresponding information from both modalities must be present in the same domain. For image-guided interventional procedures, X-ray fluoroscopy has proven to be the modality of choice. Synthesizing one modality from another in this case is an ill-posed problem due to ambiguous signal and overlapping structures in projective geometry. To take on these challenges, we present a learning-based solution to MR to X-ray projection-to-projection translation. We propose an image generator network that focuses on high representation capacity in higher resolution layers to allow for accurate synthesis of fine details in the projection images. Additionally, a weighting scheme in the loss computation that favors high-frequency structures is proposed to focus on the important details and contours in projection imaging. The proposed extensions prove valuable in generating X-ray projection images with natural appearance. Our approach achieves a deviation from the ground truth of only 6% and structural similarity measure of 0.913 ± 0.005. In particular the high frequency weighting assists in generating projection images with sharp appearance and reduces erroneously synthesized fine details.