1
|
Lombardo E, Liu PZY, Waddington DEJ, Grover J, Whelan B, Wong E, Reiner M, Corradini S, Belka C, Riboldi M, Kurz C, Landry G, Keall PJ. Experimental comparison of linear regression and LSTM motion prediction models for MLC-tracking on an MRI-linac. Med Phys 2023; 50:7083-7092. [PMID: 37782077 DOI: 10.1002/mp.16770] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/09/2023] [Revised: 08/30/2023] [Accepted: 09/17/2023] [Indexed: 10/03/2023] Open
Abstract
BACKGROUND Magnetic resonance imaging (MRI)-guided radiotherapy with multileaf collimator (MLC)-tracking is a promising technique for intra-fractional motion management, achieving high dose conformality without prolonging treatment times. To improve beam-target alignment, the geometric error due to system latency should be reduced by using temporal prediction. PURPOSE To experimentally compare linear regression (LR) and long-short-term memory (LSTM) motion prediction models for MLC-tracking on an MRI-linac using multiple patient-derived traces with different complexities. METHODS Experiments were performed on a prototype 1.0 T MRI-linac capable of MLC-tracking. A motion phantom was programmed to move a target in superior-inferior (SI) direction according to eight lung cancer patient respiratory motion traces. Target centroid positions were localized from sagittal 2D cine MRIs acquired at 4 Hz using a template matching algorithm. The centroid positions were input to one of four motion prediction models. We used (1) a LSTM network which had been optimized in a previous study on patient data from another cohort (offline LSTM). We also used (2) the same LSTM model as a starting point for continuous re-optimization of its weights during the experiment based on recent motion (offline+online LSTM). Furthermore, we implemented (3) a continuously updated LR model, which was solely based on recent motion (online LR). Finally, we used (4) the last available target centroid without any changes as a baseline (no-predictor). The predictions of the models were used to shift the MLC aperture in real-time. An electronic portal imaging device (EPID) was used to visualize the target and MLC aperture during the experiments. Based on the EPID frames, the root-mean-square error (RMSE) between the target and the MLC aperture positions was used to assess the performance of the different motion predictors. Each combination of motion trace and prediction model was repeated twice to test stability, for a total of 64 experiments. RESULTS The end-to-end latency of the system was measured to be (389 ± 15) ms and was successfully mitigated by both LR and LSTM models. The offline+online LSTM was found to outperform the other models for all investigated motion traces. It obtained a median RMSE over all traces of (2.8 ± 1.3) mm, compared to the (3.2 ± 1.9) mm of the offline LSTM, the (3.3 ± 1.4) mm of the online LR and the (4.4 ± 2.4) mm when using the no-predictor. According to statistical tests, differences were significant (p-value <0.05) among all models in a pair-wise comparison, but for the offline LSTM and online LR pair. The offline+online LSTM was found to be more reproducible than the offline LSTM and the online LR with a maximum deviation in RMSE between two measurements of 10%. CONCLUSIONS This study represents the first experimental comparison of different prediction models for MRI-guided MLC-tracking using several patient-derived respiratory motion traces. We have shown that among the investigated models, continuously re-optimized LSTM networks are the most promising to account for the end-to-end system latency in MRI-guided radiotherapy with MLC-tracking.
Collapse
Affiliation(s)
- Elia Lombardo
- Department of Radiation Oncology, LMU University Hospital, LMU Munich, Munich, Germany
| | - Paul Z Y Liu
- Image X Institute, University of Sydney Central Clinical School, Sydney, New South Wales, Australia
- Department of Medical Physics, Ingham Institute of Applied Medical Research, Liverpool, New South Wales, Australia
| | - David E J Waddington
- Image X Institute, University of Sydney Central Clinical School, Sydney, New South Wales, Australia
- Department of Medical Physics, Ingham Institute of Applied Medical Research, Liverpool, New South Wales, Australia
| | - James Grover
- Image X Institute, University of Sydney Central Clinical School, Sydney, New South Wales, Australia
- Department of Medical Physics, Ingham Institute of Applied Medical Research, Liverpool, New South Wales, Australia
| | - Brendan Whelan
- Image X Institute, University of Sydney Central Clinical School, Sydney, New South Wales, Australia
- Department of Medical Physics, Ingham Institute of Applied Medical Research, Liverpool, New South Wales, Australia
| | - Esther Wong
- Department of Medical Physics, Ingham Institute of Applied Medical Research, Liverpool, New South Wales, Australia
| | - Michael Reiner
- Department of Radiation Oncology, LMU University Hospital, LMU Munich, Munich, Germany
| | - Stefanie Corradini
- Department of Radiation Oncology, LMU University Hospital, LMU Munich, Munich, Germany
| | - Claus Belka
- Department of Radiation Oncology, LMU University Hospital, LMU Munich, Munich, Germany
- German Cancer Consortium (DKTK), partner site Munich, a partnership between DKFZ and LMU University Hospital Munich, Munich, Germany
- Bavarian Cancer Research Center (BZKF), Munich, Germany
| | - Marco Riboldi
- Department of Medical Physics, Faculty of Physics, Ludwig-Maximilians-Universität München, Garching, Germany
| | - Christopher Kurz
- Department of Radiation Oncology, LMU University Hospital, LMU Munich, Munich, Germany
| | - Guillaume Landry
- Department of Radiation Oncology, LMU University Hospital, LMU Munich, Munich, Germany
| | - Paul J Keall
- Image X Institute, University of Sydney Central Clinical School, Sydney, New South Wales, Australia
- Department of Medical Physics, Ingham Institute of Applied Medical Research, Liverpool, New South Wales, Australia
| |
Collapse
|
2
|
Lombardo E, Rabe M, Xiong Y, Nierer L, Cusumano D, Placidi L, Boldrini L, Corradini S, Niyazi M, Reiner M, Belka C, Kurz C, Riboldi M, Landry G. Evaluation of real-time tumor contour prediction using LSTM networks for MR-guided radiotherapy. Radiother Oncol 2023; 182:109555. [PMID: 36813166 DOI: 10.1016/j.radonc.2023.109555] [Citation(s) in RCA: 5] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/26/2022] [Revised: 01/24/2023] [Accepted: 02/05/2023] [Indexed: 02/24/2023]
Abstract
BACKGROUND AND PURPOSE Magnetic resonance imaging guided radiotherapy (MRgRT) with deformable multileaf collimator (MLC) tracking would allow to tackle both rigid displacement and tumor deformation without prolonging treatment. However, the system latency must be accounted for by predicting future tumor contours in real-time. We compared the performance of three artificial intelligence (AI) algorithms based on long short-term memory (LSTM) modules for the prediction of 2D-contours 500ms into the future. MATERIALS AND METHODS Models were trained (52 patients, 3.1h of motion), validated (18 patients, 0.6h) and tested (18 patients, 1.1h) with cine MRs from patients treated at one institution. Additionally, we used three patients (2.9h) treated at another institution as second testing set. We implemented 1) a classical LSTM network (LSTM-shift) predicting tumor centroid positions in superior-inferior and anterior-posterior direction which are used to shift the last observed tumor contour. The LSTM-shift model was optimized both in an offline and online fashion. We also implemented 2) a convolutional LSTM model (ConvLSTM) to directly predict future tumor contours and 3) a convolutional LSTM combined with spatial transformer layers (ConvLSTM-STL) to predict displacement fields used to warp the last tumor contour. RESULTS The online LSTM-shift model was found to perform slightly better than the offline LSTM-shift and significantly better than the ConvLSTM and ConvLSTM-STL. It achieved a 50% Hausdorff distance of 1.2mm and 1.0mm for the two testing sets, respectively. Larger motion ranges were found to lead to more substantial performance differences across the models. CONCLUSION LSTM networks predicting future centroids and shifting the last tumor contour are the most suitable for tumor contour prediction. The obtained accuracy would allow to reduce residual tracking errors during MRgRT with deformable MLC-tracking.
Collapse
Affiliation(s)
- Elia Lombardo
- Department of Radiation Oncology, University Hospital, LMU Munich, Munich 81377, Germany
| | - Moritz Rabe
- Department of Radiation Oncology, University Hospital, LMU Munich, Munich 81377, Germany
| | - Yuqing Xiong
- Department of Radiation Oncology, University Hospital, LMU Munich, Munich 81377, Germany
| | - Lukas Nierer
- Department of Radiation Oncology, University Hospital, LMU Munich, Munich 81377, Germany
| | - Davide Cusumano
- Fondazione Policlinico Universitario Agostino Gemelli IRCCS, Rome 00168, Italy
| | - Lorenzo Placidi
- Fondazione Policlinico Universitario Agostino Gemelli IRCCS, Rome 00168, Italy
| | - Luca Boldrini
- Fondazione Policlinico Universitario Agostino Gemelli IRCCS, Rome 00168, Italy
| | - Stefanie Corradini
- Department of Radiation Oncology, University Hospital, LMU Munich, Munich 81377, Germany
| | - Maximilian Niyazi
- Department of Radiation Oncology, University Hospital, LMU Munich, Munich 81377, Germany
| | - Michael Reiner
- Department of Radiation Oncology, University Hospital, LMU Munich, Munich 81377, Germany
| | - Claus Belka
- Department of Radiation Oncology, University Hospital, LMU Munich, Munich 81377, Germany; German Cancer Consortium (DKTK), Munich 81377, Germany
| | - Christopher Kurz
- Department of Radiation Oncology, University Hospital, LMU Munich, Munich 81377, Germany
| | - Marco Riboldi
- Department of Medical Physics, Faculty of Physics, Ludwig-Maximilians-Universität München, Garching b. München 85748, Germany
| | - Guillaume Landry
- Department of Radiation Oncology, University Hospital, LMU Munich, Munich 81377, Germany.
| |
Collapse
|
3
|
Syben C, Stimpel B, Roser P, Dorfler A, Maier A. Known Operator Learning Enables Constrained Projection Geometry Conversion: Parallel to Cone-Beam for Hybrid MR/X-Ray Imaging. IEEE TRANSACTIONS ON MEDICAL IMAGING 2020; 39:3488-3498. [PMID: 32746099 DOI: 10.1109/tmi.2020.2998179] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/11/2023]
Abstract
X-ray imaging is a wide-spread real-time imaging technique. Magnetic Resonance Imaging (MRI) offers a multitude of contrasts that offer improved guidance to interventionalists. As such simultaneous real-time acquisition and overlay would be highly favorable for image-guided interventions, e.g., in stroke therapy. One major obstacle in this setting is the fundamentally different acquisition geometry. MRI k -space sampling is associated with parallel projection geometry, while the X-ray acquisition results in perspective distorted projections. The classical rebinning methods to overcome this limitation inherently suffers from a loss of resolution. To counter this problem, we present a novel rebinning algorithm for parallel to cone-beam conversion. We derive a rebinning formula that is then used to find an appropriate deep neural network architecture. Following the known operator learning paradigm, the novel algorithm is mapped to a neural network with differentiable projection operators enabling data-driven learning of the remaining unknown operators. The evaluation aims in two directions: First, we give a profound analysis of the different hypotheses to the unknown operator and investigate the influence of numerical training data. Second, we evaluate the performance of the proposed method against the classical rebinning approach. We demonstrate that the derived network achieves better results than the baseline method and that such operators can be trained with simulated data without losing their generality making them applicable to real data without the need for retraining or transfer learning.
Collapse
|
4
|
Kurz C, Buizza G, Landry G, Kamp F, Rabe M, Paganelli C, Baroni G, Reiner M, Keall PJ, van den Berg CAT, Riboldi M. Medical physics challenges in clinical MR-guided radiotherapy. Radiat Oncol 2020; 15:93. [PMID: 32370788 PMCID: PMC7201982 DOI: 10.1186/s13014-020-01524-4] [Citation(s) in RCA: 79] [Impact Index Per Article: 19.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/01/2019] [Accepted: 03/24/2020] [Indexed: 12/18/2022] Open
Abstract
The integration of magnetic resonance imaging (MRI) for guidance in external beam radiotherapy has faced significant research and development efforts in recent years. The current availability of linear accelerators with an embedded MRI unit, providing volumetric imaging at excellent soft tissue contrast, is expected to provide novel possibilities in the implementation of image-guided adaptive radiotherapy (IGART) protocols. This study reviews open medical physics issues in MR-guided radiotherapy (MRgRT) implementation, with a focus on current approaches and on the potential for innovation in IGART.Daily imaging in MRgRT provides the ability to visualize the static anatomy, to capture internal tumor motion and to extract quantitative image features for treatment verification and monitoring. Those capabilities enable the use of treatment adaptation, with potential benefits in terms of personalized medicine. The use of online MRI requires dedicated efforts to perform accurate dose measurements and calculations, due to the presence of magnetic fields. Likewise, MRgRT requires dedicated quality assurance (QA) protocols for safe clinical implementation.Reaction to anatomical changes in MRgRT, as visualized on daily images, demands for treatment adaptation concepts, with stringent requirements in terms of fast and accurate validation before the treatment fraction can be delivered. This entails specific challenges in terms of treatment workflow optimization, QA, and verification of the expected delivered dose while the patient is in treatment position. Those challenges require specialized medical physics developments towards the aim of fully exploiting MRI capabilities. Conversely, the use of MRgRT allows for higher confidence in tumor targeting and organs-at-risk (OAR) sparing.The systematic use of MRgRT brings the possibility of leveraging IGART methods for the optimization of tumor targeting and quantitative treatment verification. Although several challenges exist, the intrinsic benefits of MRgRT will provide a deeper understanding of dose delivery effects on an individual basis, with the potential for further treatment personalization.
Collapse
Affiliation(s)
- Christopher Kurz
- Department of Radiation Oncology, University Hospital, LMU Munich, Marchioninistraße 15, 81377, Munich, Germany
- Department of Medical Physics, Ludwig-Maximilians-Universität München, Am Coulombwall 1, 85748, Garching, Germany
| | - Giulia Buizza
- Department of Electronics, Information and Bioengineering, Politecnico di Milano, P.za Leonardo da Vinci 32, 20133, Milano, Italy
| | - Guillaume Landry
- Department of Radiation Oncology, University Hospital, LMU Munich, Marchioninistraße 15, 81377, Munich, Germany
- Department of Medical Physics, Ludwig-Maximilians-Universität München, Am Coulombwall 1, 85748, Garching, Germany
- German Cancer Consortium (DKTK), 81377, Munich, Germany
| | - Florian Kamp
- Department of Radiation Oncology, University Hospital, LMU Munich, Marchioninistraße 15, 81377, Munich, Germany
| | - Moritz Rabe
- Department of Radiation Oncology, University Hospital, LMU Munich, Marchioninistraße 15, 81377, Munich, Germany
| | - Chiara Paganelli
- Department of Electronics, Information and Bioengineering, Politecnico di Milano, P.za Leonardo da Vinci 32, 20133, Milano, Italy
| | - Guido Baroni
- Department of Electronics, Information and Bioengineering, Politecnico di Milano, P.za Leonardo da Vinci 32, 20133, Milano, Italy
- Bioengineering Unit, National Center of Oncological Hadrontherapy (CNAO), Strada Privata Campeggi 53, 27100, Pavia, Italy
| | - Michael Reiner
- Department of Radiation Oncology, University Hospital, LMU Munich, Marchioninistraße 15, 81377, Munich, Germany
| | - Paul J Keall
- ACRF Image X Institute, University of Sydney, Sydney, NSW, 2006, Australia
| | - Cornelis A T van den Berg
- Department of Radiotherapy, University Medical Centre Utrecht, PO box 85500, 3508 GA, Utrecht, The Netherlands
| | - Marco Riboldi
- Department of Medical Physics, Ludwig-Maximilians-Universität München, Am Coulombwall 1, 85748, Garching, Germany.
| |
Collapse
|
5
|
Stimpel B, Syben C, Würfl T, Breininger K, Hoelter P, Dörfler A, Maier A. Projection-to-Projection Translation for Hybrid X-ray and Magnetic Resonance Imaging. Sci Rep 2019; 9:18814. [PMID: 31827155 PMCID: PMC6906424 DOI: 10.1038/s41598-019-55108-8] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/07/2019] [Accepted: 11/25/2019] [Indexed: 11/09/2022] Open
Abstract
Hybrid X-ray and magnetic resonance (MR) imaging promises large potential in interventional medical imaging applications due to the broad variety of contrast of MRI combined with fast imaging of X-ray-based modalities. To fully utilize the potential of the vast amount of existing image enhancement techniques, the corresponding information from both modalities must be present in the same domain. For image-guided interventional procedures, X-ray fluoroscopy has proven to be the modality of choice. Synthesizing one modality from another in this case is an ill-posed problem due to ambiguous signal and overlapping structures in projective geometry. To take on these challenges, we present a learning-based solution to MR to X-ray projection-to-projection translation. We propose an image generator network that focuses on high representation capacity in higher resolution layers to allow for accurate synthesis of fine details in the projection images. Additionally, a weighting scheme in the loss computation that favors high-frequency structures is proposed to focus on the important details and contours in projection imaging. The proposed extensions prove valuable in generating X-ray projection images with natural appearance. Our approach achieves a deviation from the ground truth of only 6% and structural similarity measure of 0.913 ± 0.005. In particular the high frequency weighting assists in generating projection images with sharp appearance and reduces erroneously synthesized fine details.
Collapse
Affiliation(s)
- Bernhard Stimpel
- Pattern Recognition Lab, Friedrich-Alexander University Erlangen-Nürnberg, Erlangen, Germany.
- Department of Neuroradiology, University Hospital Erlangen, Erlangen, Germany.
| | - Christopher Syben
- Pattern Recognition Lab, Friedrich-Alexander University Erlangen-Nürnberg, Erlangen, Germany
- Department of Neuroradiology, University Hospital Erlangen, Erlangen, Germany
| | - Tobias Würfl
- Pattern Recognition Lab, Friedrich-Alexander University Erlangen-Nürnberg, Erlangen, Germany
| | - Katharina Breininger
- Pattern Recognition Lab, Friedrich-Alexander University Erlangen-Nürnberg, Erlangen, Germany
| | - Philip Hoelter
- Department of Neuroradiology, University Hospital Erlangen, Erlangen, Germany
| | - Arnd Dörfler
- Department of Neuroradiology, University Hospital Erlangen, Erlangen, Germany
| | - Andreas Maier
- Pattern Recognition Lab, Friedrich-Alexander University Erlangen-Nürnberg, Erlangen, Germany
| |
Collapse
|