Tracking tumor boundary in MV-EPID images without implanted markers: A feasibility study

Simultaneous object detection and segmentation for patient-specific markerless lung tumor tracking in simulated radiographs with deep learning

BACKGROUND

Real-time tumor tracking is one motion management method to address motion-induced uncertainty. To date, fiducial markers are often required to reliably track lung tumors with X-ray imaging, which carries risks of complications and leads to prolonged treatment time. A markerless tracking approach is thus desirable. Deep learning-based approaches have shown promise for markerless tracking, but systematic evaluation and procedures to investigate applicability in individual cases are missing. Moreover, few efforts have been made to provide bounding box prediction and mask segmentation simultaneously, which could allow either rigid or deformable multi-leaf collimator tracking.

PURPOSE

The purpose of this study was to implement a deep learning-based markerless lung tumor tracking model exploiting patient-specific training which outputs both a bounding box and a mask segmentation simultaneously. We also aimed to compare the two kinds of predictions and to implement a specific procedure to understand the feasibility of markerless tracking on individual cases.

METHODS

We first trained a Retina U-Net baseline model on digitally reconstructed radiographs (DRRs) generated from a public dataset containing 875 CT scans and corresponding lung nodule annotations. Afterwards, we used an independent cohort of 97 lung patients to develop a patient-specific refinement procedure. In order to determine the optimal hyperparameters for automatic patient-specific training, we selected 13 patients for validation where the baseline model predicted a bounding box on planning CT (PCT)-DRR with intersection over union (IoU) with the ground-truth higher than 0.7. The final test set contained the remaining 84 patients with varying PCT-DRR IoU. For each testing patient, the baseline model was refined on the PCT-DRR to generate a patient-specific model, which was then tested on a separate 10-phase 4DCT-DRR to mimic the intrafraction motion during treatment. A template matching algorithm served as benchmark model. The testing results were evaluated by four metrics: the center of mass (COM) error and the Dice similarity coefficient (DSC) for segmentation masks, and the center of box (COB) error and the DSC for bounding box detections. Performance was compared to the benchmark model including statistical testing for significance.

RESULTS

A PCT-DRR IoU value of 0.2 was shown to be the threshold dividing inconsistent (68%) and consistent (100%) success (defined as mean bounding box DSC > 0.6) of PS models on 4DCT-DRRs. Thirty-seven out of the eighty-four testing cases had a PCT-DRR IoU above 0.2. For these 37 cases, the mean COM error was 2.6 mm, the mean segmentation DSC was 0.78, the mean COB error was 2.7 mm, and the mean box DSC was 0.83. Including the validation cases, the model was applicable to 50 out of 97 patients when using the PCT-DRR IoU threshold of 0.2. The inference time per frame was 170 ms. The model outperformed the benchmark model on all metrics, and the comparison was significant (p < 0.001) over the 37 PCT-DRR IoU > 0.2 cases, but not over the undifferentiated 84 testing cases.

CONCLUSIONS

The implemented patient-specific refinement approach based on a pre-trained baseline model was shown to be applicable to markerless tumor tracking in simulated radiographs for lung cases.

Improved Tumor Image Estimation in X-Ray Fluoroscopic Images by Augmenting 4DCT Data for Radiotherapy

Measurement of tumor position is important for the radiotherapy of lung tumors with respiratory motion. Although tumors can be observed using X-ray fluoroscopy during radiotherapy, it is often difficult to measure tumor position from X-ray image sequences accurately because of overlapping organs. To measure tumor position accurately, a method for extracting tumor intensities from X-ray image sequences using a hidden Markov model (HMM) has been proposed. However, the performance of tumor intensity extraction depends on limited knowledge regarding the tumor motion observed in the four-dimensional computed tomography (4DCT) data used to construct the HMM. In this study, we attempted to improve the performance of tumor intensity extraction by augmenting 4DCT data. The proposed method was tested using simulated datasets of X-ray image sequences. The experimental results indicated that the HMM using the augmentation method could improve tumor-tracking performance when the range of tumor movement during treatment differed from that in the 4DCT data.

Prediction of target position from multiple fiducial markers by partial least squares regression in real-time tumor-tracking radiation therapy

The purpose of this work is to show the usefulness of a prediction method of tumor location based on partial least squares regression (PLSR) using multiple fiducial markers. The trajectory data of respiratory motion of four internal fiducial markers inserted in lungs were used for the analysis. The position of one of the four markers was assumed to be the tumor position and was predicted by other three fiducial markers. Regression coefficients for prediction of the position of the tumor-assumed marker from the fiducial markers' positions is derived by PLSR. The tracking error and the gating error were evaluated assuming two possible variations. First, the variation of the position definition of the tumor and the markers on treatment planning computed tomograhy (CT) images. Second, the intra-fractional anatomical variation which leads the distance change between the tumor and markers during the course of treatment. For comparison, rigid predictions and ordinally multiple linear regression (MLR) predictions were also evaluated. The tracking and gating errors of PLSR prediction were smaller than those of other prediction methods. Ninety-fifth percentile of tracking/gating error in all trials were 3.7/4.1 mm, respectively in PLSR prediction for superior-inferior direction. The results suggested that PLSR prediction was robust to variations, and clinically applicable accuracy could be achievable for targeting tumors.

Technical Note: Synthetic treatment beam imaging for motion monitoring during spine SBRT treatments - a phantom study

PURPOSE

One of the biggest challenges in applying megavoltage (MV) treatment beam imaging for monitoring spine motion in stereotactic body radiotherapy (SBRT) is the small beam apertures in the images due to strong beam modulations in IMRT planning. The purpose of this study is to investigate the feasibility of a markerless motion tracking method in spine SBRT delivery using a novel enhanced synthetic treatment beam (ESTB) imaging technique.

METHODS

Three clinical spine SBRT plans using 6XFFF beams and sliding window IMRT technique were transferred to a thorax phantom and delivered by a TrueBeam machine. Before delivery, the phantom was aligned to the plan isocenter using CBCT setup and verified with a second CBCT, and then, 2 mm shifts were introduced in both the craniocaudal (CC) and the left-right (LR) directions with the couch. During beam delivery, MV images were continuously taken with an electronic portal imaging device (EPID) and automatically grabbed by Varian iTools Capture software with a frame rate of 11.6 Hz. After preprocessing for scatter correction and beam intensity compensation, every 50 frames of MV images were combined to generate a series of ESTB images for each beam. The ESTB images were then registered to the projections of the verification CBCT at the matched beam angles to detect the 2 mm shifts.

RESULTS

Compared to snapshot MV images, the ESTB images had significantly enlarged fields of view (FOVs) and improved image quality. Based on two-dimensional (2D) rigid registration, the ESTB image to CBCT projection matching showed submillimeter accuracy in detecting motion. Specifically, the root mean square errors in detecting the LR/CC shifts were 0.35/0.28, 0.32/0.35, 0.63/0.44, 0.55/0.51, and 0.69/0.42 mm at gantry angles 180, 160, 140, 120, and 100, respectively.

CONCLUSION

Our results in the phantom study suggest that ESTB images from a sliding window IMRT plan can be used to detect spine motion, with submillimeter precision in the 2D plane perpendicular to the beam.

A machine learning-based real-time tumor tracking system for fluoroscopic gating of lung radiotherapy

To improve respiratory-gated radiotherapy accuracy, we developed a machine learning approach for markerless tumor tracking and evaluated it using lung cancer patient data. Digitally reconstructed radiography (DRR) datasets were generated using planning 4DCT data. Tumor positions were selected on respective DRR images to place the GTV center of gravity in the center of each DRR. DRR subimages around the tumor regions were cropped so that the subimage size was defined by tumor size. Training data were then classified into two groups: positive (including tumor) and negative (not including tumor) samples. Machine learning parameters were optimized by the extremely randomized tree method. For the tracking stage, a machine learning algorithm was generated to provide a tumor likelihood map using fluoroscopic images. Prior probability tumor positions were also calculated using the previous two frames. Tumor position was then estimated by calculating maximum probability on the tumor likelihood map and prior probability tumor positions. We acquired treatment planning 4DCT images in eight patients. Digital fluoroscopic imaging systems on either side of the vertical irradiation port allowed fluoroscopic image acquisition during treatment delivery. Each fluoroscopic dataset was acquired at 15 frames per second. We evaluated the tracking accuracy and computation times. Tracking positional accuracy averaged over all patients was 1.03 ± 0.34 mm (mean ± standard deviation, Euclidean distance) and 1.76 ± 0.71 mm ([Formula: see text] percentile). Computation time was 28.66 ± 1.89 ms/frame averaged over all frames. Our markerless algorithm successfully estimated tumor position in real time.

Technical Note: 3D localization of lung tumors on cone beam CT projections via a convolutional recurrent neural network

PURPOSE

To design a convolutional recurrent neural network (CRNN) that calculates three-dimensional (3D) positions of lung tumors from continuously acquired cone beam computed tomography (CBCT) projections, and facilitates the sorting and reconstruction of 4D-CBCT images.

METHOD

Under an IRB-approved clinical lung protocol, kilovoltage (kV) projections of the setup CBCT were collected in free-breathing. Concurrently, an electromagnetic signal-guided system recorded motion traces of three transponders implanted in or near the tumor. Convolutional recurrent neural network was designed to utilize a convolutional neural network (CNN) for extracting relevant features of the kV projections around the tumor, followed by a recurrent neural network for analyzing the temporal patterns of the moving features. Convolutional recurrent neural network was trained on the simultaneously collected kV projections and motion traces, subsequently utilized to calculate motion traces solely based on the continuous feed of kV projections. To enhance performance, CRNN was also facilitated by frequent calibrations (e.g., at 10° gantry rotation intervals) derived from cross-correlation-based registrations between kV projections and templates created from the planning 4DCT. Convolutional recurrent neural network was validated on a leave-one-out strategy using data from 11 lung patients, including 5500 kV images. The root-mean-square error between the CRNN and motion traces was calculated to evaluate the localization accuracy.

RESULT

Three-dimensional displacement around the simulation position shown in the Calypso traces was 3.4 ± 1.7 mm. Using motion traces as ground truth, the 3D localization error of CRNN with calibrations was 1.3 ± 1.4 mm. CRNN had a success rate of 86 ± 8% in determining whether the motion was within a 3D displacement window of 2 mm. The latency was 20 ms when CRNN ran on a high-performance computer cluster.

CONCLUSIONS

CRNN is able to provide accurate localization of lung tumors with aid from frequent recalibrations using the conventional cross-correlation-based registration approach, and has the potential to remove reliance on the implanted fiducials.

A real-time, single-exposure, dual-energy subtraction mask for markerless tumor tracking in radiotherapy: Proof of concept

INTRODUCTION

Bone density can interfere with fluoroscopy-guided tumor tracking in radiotherapy. To improve markerless tumor tracking accuracy, we developed a dual energy subtraction (DES) mechanical image mask to use with a single x-ray exposure.

METHODS

The DES mask consists of 2-mm-thick stainless-steel with 128 pairs of slits (0.388 mm width and openings), designed to match the dynamic flat panel detector (DFPD) pixel size. This was set on the front of the DFPD. This results in a DFPD image with one containing the exposed pixels and one containing the masked pixels. The masked pixel columns were interpolated from adjacent pixels and a subtraction image was generated from the interpolated images to make a bone suppression (BS) image. A chest phantom was set on the commercially available moving table (CIRS DYNAMIC PLATFORM 008PL) and DFPD images were acquired. A reference BS image was generated by double-exposure DES with and without a 2-mm-thick stainless-steel plate. Image quality and markerless tumor tracking accuracy were then evaluated.

RESULTS

The DES mask decreased most of the visible bone densities from the chest phantom image acquired with a single exposure for a peak-signal-to-noise-ratio/structural similarity index measure (PSNR/SSIM) of 25.3 db/0.685). The tracking positional error, originally 12.6 mm, was improved to 0.2 mm.

CONCLUSIONS

The DES mask can aid in BS image on fluoroscopic imaging and may be useful in markerless tumor tracking.

Markerless Pancreatic Tumor Target Localization Enabled By Deep Learning

PURPOSE

Deep learning is an emerging technique that allows us to capture imaging information beyond the visually recognizable level of a human being. Because of the anatomic characteristics and location, on-board target verification for radiation delivery to pancreatic tumors is a challenging task. Our goal was to use a deep neural network to localize the pancreatic tumor target on kV x-ray images acquired using an on-board imager for image guided radiation therapy.

METHODS AND MATERIALS

The network is set up in such a way that the input is either a digitally reconstructed radiograph image or a monoscopic x-ray projection image acquired by the on-board imager from a given direction, and the output is the location of the planning target volume in the projection image. To produce a sufficient number of training x-ray images reflecting the vast number of possible clinical scenarios of anatomy distribution, a series of changes were introduced to the planning computed tomography images, including deformation, rotation, and translation, to simulate inter- and intrafractional variations. After model training, the accuracy of the model was evaluated by retrospectively studying patients who underwent pancreatic cancer radiation therapy. Statistical analysis using mean absolute differences (MADs) and Lin's concordance correlation coefficient were used to assess the accuracy of the predicted target positions.

RESULTS

MADs between the model-predicted and the actual positions were found to be less than 2.60 mm in anteroposterior, lateral, and oblique directions for both axes in the detector plane. For comparison studies with and without fiducials, MADs are less than 2.49 mm. For all cases, Lin's concordance correlation coefficients between the predicted and actual positions were found to be better than 93%, demonstrating the success of the proposed deep learning for image guided radiation therapy.

CONCLUSIONS

We demonstrated that markerless pancreatic tumor target localization is achievable with high accuracy by using a deep learning technique approach.

A Novel Markerless Lung Tumor-Tracking Method Using Treatment MV Beam Imaging

A novel method was developed to track lung tumor motion in real time during radiation therapy with the purpose to allow target radiation dose escalation while simultaneously reducing the dose to sensitive structures, thereby increasing local control without increasing toxicity. This method analyzes beam’s eye view radiation therapy treatment megavoltage (MV) images with simulated digitally reconstructed radiographs (DRRs) as references. Instead of comparing global DRRs with projection images, this method incorporates a technique that divides the global composite DRR and the corresponding MV projection into sub-images called tiles. Registration is performed independently on tile pairs in order to reduce the effects of global discrepancies due to scattering or imaging modality differences. This algorithm was evaluated by phantom studies while simulated tumors were controlled to move with various patterns in a complex humanoid torso. Approximately 15,000 phantom MV images were acquired at nine gantry angles, with different tumors moving within ranges between 10 and 20 mm. Tumors were successfully identified on every projection with a total maximum/average error of 1.84/0.98 mm. This algorithm was also applied to over 5,000 frames of MV projections acquired during radiation therapy of five lung cancer patients. This tumor-tracking methodology is capable of accurately locating lung tumors during treatment without implanting any internal fiducial markers nor delivering extra imaging radiation doses.

Enhancement of megavoltage electronic portal images for markerless tumor tracking

Purpose

The poor quality of megavoltage (MV) images from electronic portal imaging device (EPID) hinders visual verification of tumor targeting accuracy particularly during markerless tumor tracking. The aim of this study was to investigate the effect of a few representative image processing treatments on visual verification and detection capability of tumors under auto tracking.

Methods

Images of QC‐3 quality phantom, a single patient's setup image, and cine images of two‐lung cancer patients were acquired. Three image processing methods were individually employed to the same original images. For each deblurring, contrast enhancement, and denoising, a total variation deconvolution, contrast‐limited adaptive histogram equalization (CLAHE), and median filter were adopted, respectively. To study the effect of image enhancement on tumor auto‐detection, a tumor tracking algorithm was adopted in which the tumor position was determined as the minimum point of the mean of the sum of squared pixel differences (MSSD) between two images. The detectability and accuracy were compared.

Results

Deblurring of a quality phantom image yielded sharper edges, while the contrast‐enhanced image was more readable with improved structural differentiation. Meanwhile, the denoising operation resulted in noise reduction, however, at the cost of sharpness. Based on comparison of pixel value profiles, contrast enhancement outperformed others in image perception. During the tracking experiment, only contrast enhancement resulted in tumor detection in all images using our tracking algorithm. Deblurring failed to determine the target position in two frames out of a total of 75 images. For original and denoised set, target location was not determined for the same five images. Meanwhile, deblurred image showed increased detection accuracy compared with the original set. The denoised image resulted in decreased accuracy. In the case of contrast‐improved set, the tracking accuracy was nearly maintained as that of the original image.

Conclusions

Considering the effect of each processing on tumor tracking and the visual perception in a limited time, contrast enhancement would be the first consideration to visually verify the tracking accuracy of tumors on MV EPID without sacrificing tumor detectability and detection accuracy.

Novel real-time tumor-contouring method using deep learning to prevent mistracking in X-ray fluoroscopy

Robustness to obstacles is the most important factor necessary to achieve accurate tumor tracking without fiducial markers. Some high-density structures, such as bone, are enhanced on X-ray fluoroscopic images, which cause tumor mistracking. Tumor tracking should be performed by controlling "importance recognition": the understanding that soft-tissue is an important tracking feature and bone structure is unimportant. We propose a new real-time tumor-contouring method that uses deep learning with importance recognition control. The novelty of the proposed method is the combination of the devised random overlay method and supervised deep learning to induce the recognition of structures in tumor contouring as important or unimportant. This method can be used for tumor contouring because it uses deep learning to perform image segmentation. Our results from a simulated fluoroscopy model showed accurate tracking of a low-visibility tumor with an error of approximately 1 mm, even if enhanced bone structure acted as an obstacle. A high similarity of approximately 0.95 on the Jaccard index was observed between the segmented and ground truth tumor regions. A short processing time of 25 ms was achieved. The results of this simulated fluoroscopy model support the feasibility of robust real-time tumor contouring with fluoroscopy. Further studies using clinical fluoroscopy are highly anticipated.

Dosimetric evaluation of MLC-based dynamic tumor tracking radiotherapy using digital phantom: Desired setup margin for tracking radiotherapy

The purpose of this study is to evaluate the dosimetric impact of the margin on the multileaf collimator-based dynamic tumor tracking plan. Furthermore, an equivalent setup margin (EM) of the tracking plan was determined according to the gated plan. A 4-dimensional extended cardiac-torso was used to create 9 digital phantom datasets of different tumor diameters (TDs) of 1, 3, and 5 cm and motion ranges (MRs) of 1, 2, and 3 cm. For each dataset, respiratory gating (30% to 70% phase) and tumor tracking treatment plans were prepared using 8-field 3-dimensional conformal radiation therapy by 4-dimensional dose calculation. The total lung V20 was calculated to evaluate the dosimetric impact for each case and to estimate the EM with the same impact on lung V20 obtained with the gating plan with a setup margin of 5 mm. The EMs for {TD = 1 cm, MR = 1 cm}, {TD = 1 cm, MR = 2 cm}, and {TD = 1 cm, MR = 3 cm} were estimated as 5.00, 4.16, and 4.24 mm, respectively. The EMs for {TD = 5 cm, MR = 1 cm}, {TD = 5 cm, MR = 2 cm}, and {TD = 5 cm, MR = 3 cm} were estimated as 4.24 mm, 6.35 mm, and 7.49 mm, respectively. This result showed that with a larger MR, the EM was found to be increased. In addition, with a larger TD, the EM became smaller. Our result showing the EMs provided the desired accuracy for multileaf collimator-based dynamic tumor tracking radiotherapy.