Technical Note: 3D localization of lung tumors on cone beam CT projections via a convolutional recurrent neural network

Artificial intelligence-based motion tracking in cancer radiotherapy: A review

Radiotherapy aims to deliver a prescribed dose to the tumor while sparing neighboring organs at risk (OARs). Increasingly complex treatment techniques such as volumetric modulated arc therapy (VMAT), stereotactic radiosurgery (SRS), stereotactic body radiotherapy (SBRT), and proton therapy have been developed to deliver doses more precisely to the target. While such technologies have improved dose delivery, the implementation of intra-fraction motion management to verify tumor position at the time of treatment has become increasingly relevant. Artificial intelligence (AI) has recently demonstrated great potential for real-time tracking of tumors during treatment. However, AI-based motion management faces several challenges, including bias in training data, poor transparency, difficult data collection, complex workflows and quality assurance, and limited sample sizes. This review presents the AI algorithms used for chest, abdomen, and pelvic tumor motion management/tracking for radiotherapy and provides a literature summary on the topic. We will also discuss the limitations of these AI-based studies and propose potential improvements.

Deep learning-based target decomposition for markerless lung tumor tracking in radiotherapy

BACKGROUND

In radiotherapy, real-time tumor tracking can verify tumor position during beam delivery, guide the radiation beam to target the tumor, and reduce the chance of a geometric miss. Markerless kV x-ray image-based tumor tracking is challenging due to the low tumor visibility caused by tumor-obscuring structures. Developing a new method to enhance tumor visibility for real-time tumor tracking is essential.

PURPOSE

To introduce a novel method for markerless kV image-based tracking of lung tumors via deep learning-based target decomposition.

METHODS

We utilized a conditional Generative Adversarial Network (cGAN), known as Pix2Pix, to build a patient-specific model and generate the synthetic decomposed target image (sDTI) to enhance tumor visibility on the real-time kV projection images acquired by the onboard kV imager equipped on modern linear accelerators. We used 4DCT simulation images to generate the digitally reconstructed radiograph (DRR) and DTI image pairs for model training. We augmented the training dataset by randomly shifting the 4DCT in the superior-inferior, anterior-posterior, and left-right directions during the DRR and DTI generation process. We performed real-time 2D tumor tracking via template matching between the DTI generated from the CT simulation and the sDTI generated from the real-time kV projection images. We validated the proposed method using nine patients' datasets with implanted beacons near the tumor.

RESULTS

The sDTI can effectively improve the image contrast around the lung tumors on the kV projection images for the nine patients. With the beacon motion as ground truth, the tracking errors were on average 0.8 ± 0.7 mm in the superior-inferior (SI) direction and 0.9 ± 0.8 mm in the in-plane left-right (IPLR) direction. The percentage of successful tracking, defined as a tracking error less than 2 mm in the SI direction, is 92.2% on the 4312 tested images. The patient-specific model took approximately 12 h to train. During testing, it took approximately 35 ms to generate one sDTI, and 13 ms to perform the tumor tracking using template matching.

CONCLUSIONS

Our method offers the potential solution for nearly real-time markerless lung tumor tracking. It achieved a high level of accuracy and an impressive tracking rate. Further development of 3D lung tumor tracking is warranted.

Deep learning-based target tracking with X-ray images for radiotherapy: a narrative review

Background and Objective

As one of the main treatment modalities, radiotherapy (RT) (also known as radiation therapy) plays an increasingly important role in the treatment of cancer. RT could benefit greatly from the accurate localization of the gross tumor volume and circumambient organs at risk (OARs). Modern linear accelerators (LINACs) are typically equipped with either gantry-mounted or room-mounted X-ray imaging systems, which provide possibilities for marker-less tracking with two-dimensional (2D) kV X-ray images. However, due to organ overlapping and poor soft tissue contrast, it is challenging to track the target directly and precisely with 2D kV X-ray images. With the flourishing development of deep learning in the field of image processing, it is possible to achieve real-time marker-less tracking of targets with 2D kV X-ray images in RT using advanced deep-learning frameworks. This article sought to review the current development of deep learning-based target tracking with 2D kV X-ray images and discuss the existing limitations and potential solutions. Finally, it also discusses some common challenges and potential future developments.

Methods

Manual searches of the Web of Science, and PubMed, and Google Scholar were carried out to retrieve English-language articles. The keywords used in the searches included "radiotherapy, radiation therapy, motion tracking, target tracking, motion estimation, motion monitoring, X-ray images, digitally reconstructed radiographs, deep learning, convolutional neural network, and deep neural network". Only articles that met the predetermined eligibility criteria were included in the review. Ultimately, 23 articles published between March 2019 and December 2023 were included in the review.

Key Content and Findings

In this article, we narratively reviewed deep learning-based target tracking with 2D kV X-ray images in RT. The existing limitations, common challenges, possible solutions, and future directions of deep learning-based target tracking were also discussed. The use of deep learning-based methods has been shown to be feasible in marker-less target tracking and real-time motion management. However, it is still quite challenging to directly locate tumor and OARs in real-time with 2D kV X-ray images, and more technical and clinical efforts are needed.

Conclusions

Deep learning-based target tracking with 2D kV X-ray images is a promising method in motion management during RT. It has the potential to track the target in real time, recognize motion, reduce the extended margin, and better spare the normal tissue. However, it still has many issues that demand prompt attention, and further development before it can be put into clinical practice.

Study on high-precision three-dimensional reconstruction of pulmonary lesions and surrounding blood vessels based on CT images

The adoption of computerized tomography (CT) technology has significantly elevated the role of pulmonary CT imaging in diagnosing and treating pulmonary diseases. However, challenges persist due to the complex relationship between lesions within pulmonary tissue and the surrounding blood vessels. These challenges involve achieving precise three-dimensional reconstruction while maintaining accurate relative positioning of these elements. To effectively address this issue, this study employs a semi-automatic precise labeling process for the target region. This procedure ensures a high level of consistency in the relative positions of lesions and the surrounding blood vessels. Additionally, a morphological gradient interpolation algorithm, combined with Gaussian filtering, is applied to facilitate high-precision three-dimensional reconstruction of both lesions and blood vessels. Furthermore, this technique enables post-reconstruction slicing at any layer, facilitating intuitive exploration of the correlation between blood vessels and lesion layers. Moreover, the study utilizes physiological knowledge to simulate real-world blood vessel intersections, determining the range of blood vessel branch angles and achieving seamless continuity at internal blood vessel branch points. The experimental results achieved a satisfactory reconstruction with an average Hausdorff distance of 1.5 mm and an average Dice coefficient of 92%, obtained by comparing the reconstructed shape with the original shape,the approach also achieves a high level of accuracy in three-dimensional reconstruction and visualization. In conclusion, this study is a valuable source of technical support for the diagnosis and treatment of pulmonary diseases and holds promising potential for widespread adoption in clinical practice.

Deep learning-based markerless lung tumor tracking in stereotactic radiotherapy using Siamese networks

BACKGROUND

Radiotherapy (RT) is involved in about 50% of all cancer patients, making it a very important treatment modality. The most common type of RT is external beam RT, which consists of delivering the radiation to the tumor from outside the body. One novel treatment delivery method is volumetric modulated arc therapy (VMAT), where the gantry continuously rotates around the patient during the radiation delivery.

PURPOSE

Accurate tumor position monitoring during stereotactic body radiotherapy (SBRT) for lung tumors can help to ensure that the tumor is only irradiated when it is inside the planning target volume. This can maximize tumor control and reduce uncertainty margins, lowering organ-at-risk dose. Conventional tracking methods are prone to errors, or have a low tracking rate, especially for small tumors that are in close vicinity to bony structures.

METHODS

We investigated patient-specific deep Siamese networks for real-time tumor tracking, during VMAT. Due to lack of ground truth tumor locations in the kilovoltage (kV) images, each patient-specific model was trained on synthetic data (DRRs), generated from the 4D planning CT scans, and evaluated on clinical data (x-rays). Since there are no annotated datasets with kV images, we evaluated the model on a 3D printed anthropomorphic phantom but also on six patients by computing the correlation coefficient with the breathing-related vertical displacement of the surface-mounted marker (RPM). For each patient/phantom, we used 80% of DRRs for training and 20% for validation.

RESULTS

The proposed Siamese model outperformed the conventional benchmark template matching-based method (RTR): (1) when evaluating both methods on the 3D phantom, the Siamese model obtained a 0.57-0.79-mm mean absolute distance to the ground truth tumor locations, compared to 1.04-1.56 mm obtained by RTR; (2) on patient data, the Siamese-determined longitudinal tumor position had a correlation coefficient of 0.71-0.98 with the RPM, compared to 0.07-0.85 for RTR; (3) the Siamese model had a 100% tracking rate, compared to 62%-82% for RTR.

CONCLUSIONS

Based on these results, we argue that Siamese-based real-time 2D markerless tumor tracking during radiation delivery is possible. Further investigation and development of 3D tracking is warranted.

Accelerating multi-echo chemical shift encoded water-fat MRI using model-guided deep learning

PURPOSE

To accelerate chemical shift encoded (CSE) water-fat imaging by applying a model-guided deep learning water-fat separation (MGDL-WF) framework to the undersampled k-space data.

METHODS

A model-guided deep learning water-fat separation framework is proposed for the acceleration using Cartesian/radial undersampling data. The proposed MGDL-WF combines the power of CSE water-fat imaging model and data-driven deep learning by jointly using a multi-peak fat model and a modified residual U-net network. The model is used to guide the image reconstruction, and the network is used to capture the artifacts induced by the undersampling. A data consistency layer is used in MGDL-WF to ensure the output images to be consistent with the k-space measurements. A Gauss-Newton iteration algorithm is adapted for the gradient updating of the networks.

RESULTS

Compared with the compressed sensing water-fat separation (CS-WF) algorithm/2-step procedure algorithm, the MGDL-WF increased peak signal-to-noise ratio (PSNR) by 5.31/5.23, 6.11/4.54, and 4.75 dB/1.88 dB with Cartesian sampling, and by 4.13/6.53, 2.90/4.68, and 1.68 dB/3.48 dB with radial sampling, at acceleration rates (R) of 4, 6, and 8, respectively. By using MGDL-WF, radial sampling increased the PSNR by 2.07 dB at R = 8, compared with Cartesian sampling.

CONCLUSIONS

The proposed MGDL-WF enables exploiting features of the water images and fat images from the undersampled multi-echo data, leading to improved performance in the accelerated CSE water-fat imaging. By using MGDL-WF, radial sampling can further improve the image quality with comparable scan time in comparison with Cartesian sampling.

Dosimetric Study of Deep Learning-Guided ITV Prediction in Cone-beam CT for Lung Stereotactic Body Radiotherapy

Purpose

The purpose of this study was to evaluate the accuracy of a lung stereotactic body radiotherapy (SBRT) treatment plan with the target of a newly predicted internal target volume (ITV_predict) and the feasibility of its clinical application. ITV_predict was automatically generated by our in-house deep learning model according to the cone-beam CT (CBCT) image database.

Method

A retrospective study of 45 patients who underwent SBRT was involved, and Mask R-CNN based algorithm model helped to predict the internal target volume (ITV) using the CBCT image database. The geometric accuracy of ITV_predict was verified by the Dice Similarity Coefficient (DSC), 3D Motion Range (R_3D), Relative Volume Index (RVI), and Hausdorff Distance (HD). The PTV_predict was generated by ITV_predict, which was registered and then projected on free-breath CT (FBCT) images. The PTV_FBCT was margined from the GTV on FBCT images gross tumor volume on free-breath CT (GTV_FBCT). Treatment plans with the target of Predict planning target volume on CBCT images (PTV_predict) and planning target volume on free-breath CT (PTV_FBCT) were respectively re-established, and the dosimetric parameters included the ratio of the volume of patients receiving at least the prescribed dose to the volume of PTV (R_100%), the ratio of the volume of patients receiving at least 50% of the prescribed dose to the volume of PTV in the Radiation Therapy Oncology Group (RTOG) 0813 Trial (R_50%), Gradient Index (GI), and the maximum dose 2 cm from the PTV (D_2cm), which were evaluated via Plan_4DCT, plan which based on PTV_predict (Plan_predict), and plan which based on PTV_FBCT (Plan_FBCT).

Result

The geometric results showed that there existed a good correlation between ITV_predict and ITV on the 4-dimensional CT [ITV_4DCT; DSC= 0.83 ±0.18]. However, the average volume of ITV_predict was 10% less than that of ITV_4DCT (p = 0.333). No significant difference in dose coverage was found in V_100% for the ITV with 99.98 ± 0.04% in the ITV_4DCT vs. 97.56 ± 4.71% in the ITV_predict (p = 0.162). Dosimetry parameters of PTV, including R_100%, R_50%, GI and D_2cm showed no statistically significant difference between each plan (p > 0.05).

Conclusion

Dosimetric parameters of Plan_predict are clinically comparable to those of the original Plan_4DCT. This study confirmed that the treatment plan based on ITV_predict produced by our model could automatically meet clinical requirements. Thus, for patients undergoing lung SBRT, the model has great potential for using CBCT images for ITV contouring which can be used in treatment planning.

Deep Learning-Based Internal Target Volume (ITV) Prediction Using Cone-Beam CT Images in Lung Stereotactic Body Radiotherapy

Purpose:This study aims to develop a deep learning (DL)-based (Mask R-CNN) method to predict the internal target volume (ITV) in cone beam computed tomography (CBCT) images for lung stereotactic body radiotherapy (SBRT) patients and to evaluate the prediction accuracy of the model using 4DCT as ground truth. Methods and Materials: This study enrolled 78 phantom cases and 156 patient cases who received SBRT treatment. We used a novel DL model (Mask R-CNN) to identify and delineate lung tumor ITV in CBCT images. The results of the DL-based method were compared quantitatively with the ground truth (4DCT) using 4 metrics, including Dice Similarity Coefficient (DSC), Relative Volume Index (RVI), 3D Motion Range (R3D), and Hausdorff Surface Distance (HD). Paired t-tests were used to determine the differences between the DL-based method and manual contouring. Results: The DSC value for 4DCTMIP versus CBCT is 0.86 ± 0.16 and for 4DCTAVG versus CBCT is 0.83 ± 0.18, indicating a high similarity of tumor delineation in CBCT and 4DCT. The mean Accuracy Precision (mAP), R3D, RVI, and HD values for phantom evaluation are 0.94 ± 0.04, 1.37 ± 0.36, 0.79 ± 0.02, and 6.79 ± 0.68, respectively. For patient evaluation, the mAP, R3D, RVI, and HD achieved averaged values of 0.74 ± 0.23, 2.39 ± 1.59, 1.27 ± 0.47, and 17.00 ± 19.89, respectively. These results showed a good correlation between predicted ITV and manually contoured ITV. The phantom p-value for RVI, R3D, and HD are 0.75, 0.08, 0.86, and patient p-value are 0.53, 0.07, 0.28, respectively. These p-values for phantom and patient showed no significant difference between the predicted ITV and physician's manual contouring. Conclusion:The current improved method (Mask R-CNN) yielded a good similarity between predicted ITV in CBCT and the manual contouring in 4DCT, thus indicating great potential for using CBCT for patient ITV contouring.

Improvement of kilovoltage intrafraction monitoring accuracy through gantry angles selection

Kilovoltage intrafraction monitoring (KIM) is a method allowing to precisely infer the tumour trajectory based on cone beam computed tomography (CBCT) 2D-projections. However, its accuracy is deteriorated in the case of highly mobile tumours involving hysteresis. A first adaptation of KIM consisting of a prior amplitude based binning step has been developed in order to minimize the errors of the original model (phase-KIM). In this work, we propose enhanced methods (KIM_{sub-arc optim}and phase-KIM_{sub-arc optim}) to improve the accuracy of KIM and phase-KIM which relies on the selection of the optimal starting CBCT gantry angle. Aiming at demonstrating the interest of our approach, we carried out a simulation study and an experimental study: we compared the accuracy of the conventional versus sub-arc optim methods on simulated realistic tumour motions with amplitudes ranging from 5 to 30 mm in 1 mm increments. The same approach was performed using a lung dynamic phantom generating a 30 mm amplitude sinusoidal motion. The results show that for in-silico simulated motions of 10, 20 and 30 mm amplitude, the three-dimensional root mean square error (3D-RMSE) can be reduced by 0.67 mm, 0.91 mm, 0.94 mm and 0.18 mm, 0.25 mm, 0.28 mm using KIM_{sub-arc optim}and phase-KIM_{sub-arc optim}respectively. Considering all in-silico simulated trajectories, the percentage of errors larger than 1 mm decreases from 21.9% down to 1.6% for KIM (p < 0.001) and from 6.6% down to 1.2% for phase-KIM (p < 0.001). Experimentally, the 3D-RMSE is lowered by 0.5732 mm for KIM and by 0.1 mm for phase-KIM. The percentage of errors larger than 1 mm falls from 39.7% down to 18.5% for KIM and from 23.2% down to 11.1% for phase-KIM. In conclusion, our method efficiently anticipates CBCT gantry angles associated with a significantly better accuracy by using KIM and phase-KIM.