Real-time liver tracking algorithm based on LSTM and SVR networks for use in surface-guided radiation therapy

Explicitly encoding the cyclic nature of breathing signal allows for accurate breathing motion prediction in radiotherapy with minimal training data

Background and purpose

Active breathing motion management in radiotherapy consists of motion monitoring, quantification and mitigation. It is impacted by associated latencies of a few 100 ms. Artificial neural networks can successfully predict breathing motion and eliminate latencies. However, they require usually a large dataset for training. The objective of this work was to demonstrate that explicitly encoding the cyclic nature of the breathing signal into the training data enables significant reduction of training datasets which can be obtained from healthy volunteers.

Material and methods

Seventy surface scanner breathing signals from 25 healthy volunteers in anterior-posterior direction were used for training and validation (ratio 4:1) of long short-term memory models. The model performance was compared to a model using decomposition into phase, amplitude and a time-dependent baseline. Testing of the models was performed on 55 independent breathing signals in anterior-posterior direction from surface scanner (35 lung, 20 liver) of 30 patients with a mean breathing amplitude of (5.9 ± 6.7) mm.

Results

Using the decomposed breathing signal allowed for a reduction of the absolute root-mean square error (RMSE) from 0.34 mm to 0.12 mm during validation. Testing using patient data yielded an average absolute RMSE of the breathing signal of (0.16 ± 0.11) mm with a prediction horizon of 500 ms.

Conclusion

It was demonstrated that a motion prediction model can be trained with less than 100 datasets of healthy volunteers if breathing cycle parameters are considered. Applied to 55 patients, the model predicted breathing motion with a high accuracy.

Online advance respiration prediction model for percutaneous puncture robotics

PURPOSE

Surgical robots have significant research value and clinical significance in the field of percutaneous punctures. There have been numerous studies on ultrasound-guided percutaneous surgical robots; however, addressing the respiratory compensation problem of deep punctures remains a significant obstacle. Herein we propose a robotic system for percutaneous puncture with respiratory compensation.

METHODS

We proposed an online advance respiratory prediction model based on Bidirectional Gate Recurrent Unit (Bi-GRU) for the respiratory prediction requirements of surgical robot systems. By analyzing the main factors governing the accuracy of the respiratory motion prediction models, various parameters of the online advance prediction model were optimized. Subsequently, we integrated and developed ultrasound-guided percutaneous puncture robot software and a hardware platform to implement respiratory compensation, thus verifying the effectiveness and reliability of various key technologies in the system.

RESULTS

The proposed respiratory prediction model has a significantly reduced update time, with an average root mean square error (RMSE) of less than 0.4 mm. This represents a reduction of ~ 20% compared to the online training long short-term memory(LSTM). By conducting puncture experiments based on a respiratory phantom, the average puncture error was 2.71 ± 0.65 mm and the average single-round puncture time was 65.00 ± 6.67 s.

CONCLUSION

Herein we proposed and optimized an online training respiratory prediction network model based on Bi-GRU. The stability and reliability of this system are verified by conducting puncture experiments on a respiratory phantom.

A Respiratory Motion Prediction Method Based on LSTM-AE with Attention Mechanism for Spine Surgery

Respiratory motion-induced vertebral movements can adversely impact intraoperative spine surgery, resulting in inaccurate positional information of the target region and unexpected damage during the operation. In this paper, we propose a novel deep learning architecture for respiratory motion prediction, which can adapt to different patients. The proposed method utilizes an LSTM-AE with attention mechanism network that can be trained using few-shot datasets during operation. To ensure real-time performance, a dimension reduction method based on the respiration-induced physical movement of spine vertebral bodies is introduced. The experiment collected data from prone-positioned patients under general anaesthesia to validate the prediction accuracy and time efficiency of the LSTM-AE-based motion prediction method. The experimental results demonstrate that the presented method (RMSE: 4.39%) outperforms other methods in terms of accuracy within a learning time of 2 min. The maximum predictive errors under the latency of 333 ms with respect to the x, y, and z axes of the optical camera system were 0.13, 0.07, and 0.10 mm, respectively, within a motion range of 2 mm.

Real-time liver tumor localization via combined surface imaging and a single x-ray projection

Objective. Real-time imaging, a building block of real-time adaptive radiotherapy, provides instantaneous knowledge of anatomical motion to drive delivery adaptation to improve patient safety and treatment efficacy. The temporal constraint of real-time imaging (<500 milliseconds) significantly limits the imaging signals that can be acquired, rendering volumetric imaging and 3D tumor localization extremely challenging. Real-time liver imaging is particularly difficult, compounded by the low soft tissue contrast within the liver. We proposed a deep learning (DL)-based framework (Surf-X-Bio), to track 3D liver tumor motion in real-time from combined optical surface image and a single on-board x-ray projection.Approach. Surf-X-Bio performs mesh-based deformable registration to track/localize liver tumors volumetrically via three steps. First, a DL model was built to estimate liver boundary motion from an optical surface image, using learnt motion correlations between the respiratory-induced external body surface and liver boundary. Second, the residual liver boundary motion estimation error was further corrected by a graph neural network-based DL model, using information extracted from a single x-ray projection. Finally, a biomechanical modeling-driven DL model was applied to solve the intra-liver motion for tumor localization, using the liver boundary motion derived via prior steps.Main results. Surf-X-Bio demonstrated higher accuracy and better robustness in tumor localization, as compared to surface-image-only and x-ray-only models. By Surf-X-Bio, the mean (±s.d.) 95-percentile Hausdorff distance of the liver boundary from the 'ground-truth' decreased from 9.8 (±4.5) (before motion estimation) to 2.4 (±1.6) mm. The mean (±s.d.) center-of-mass localization error of the liver tumors decreased from 8.3 (±4.8) to 1.9 (±1.6) mm.Significance. Surf-X-Bio can accurately track liver tumors from combined surface imaging and x-ray imaging. The fast computational speed (<250 milliseconds per inference) allows it to be applied clinically for real-time motion management and adaptive radiotherapy.

Respiratory motion prediction based on deep artificial neural networks in CyberKnife system: A comparative study

BACKGROUND

In external beam radiotherapy, a prediction model is required to compensate for the temporal system latency that affects the accuracy of radiation dose delivery. This study focused on a thorough comparison of seven deep artificial neural networks to propose an accurate and reliable prediction model.

METHODS

Seven deep predictor models are trained and tested with 800 breathing signals. In this regard, a nonsequential-correlated hyperparameter optimization algorithm is developed to find the best configuration of parameters for all models. The root mean square error (RMSE), mean absolute error, normalized RMSE, and statistical F-test are also used to evaluate network performance.

RESULTS

Overall, tuning the hyperparameters results in a 25%-30% improvement for all models compared to previous studies. The comparison between all models also shows that the gated recurrent unit (GRU) with RMSE = 0.108 ± 0.068 mm predicts respiratory signals with higher accuracy and better performance.

CONCLUSION

Overall, tuning the hyperparameters in the GRU model demonstrates a better result than the hybrid predictor model used in the CyberKnife VSI system to compensate for the 115 ms system latency. Additionally, it is demonstrated that the tuned parameters have a significant impact on the prediction accuracy of each model.

Comparison of initial learning algorithms for long short-term memory method on real-time respiratory signal prediction

Aim

This study aimed to examine the effect of the weight initializers on the respiratory signal prediction performance using the long short-term memory (LSTM) model.

Methods

Respiratory signals collected with the CyberKnife Synchrony device during 304 breathing motion traces were used in this study. The effectiveness of four weight initializers (Glorot, He, Orthogonal, and Narrow-normal) on the prediction performance of the LSTM model was investigated. The prediction performance was evaluated by the normalized root mean square error (NRMSE) between the ground truth and predicted respiratory signal.

Results

Among the four initializers, the He initializer showed the best performance. The mean NRMSE with 385-ms ahead time using the He initializer was superior by 7.5%, 8.3%, and 11.3% as compared to that using the Glorot, Orthogonal, and Narrow-normal initializer, respectively. The confidence interval of NRMSE using Glorot, He, Orthogonal, and Narrow-normal initializer were [0.099, 0.175], [0.097, 0.147], [0.101, 0.176], and [0.107, 0.178], respectively.

Conclusions

The experiment results in this study indicated that He could be a valuable initializer in the LSTM model for the respiratory signal prediction.

[Prediction of respiratory motion based on sequential embedding combined with relational embedding]

OBJECTIVE

To propose a deep learning model for modeling and prediction of the integration of respiratory motion in all directions.

METHODS

The respiratory motion signals in different directions were input into the sequential embedding layer composed of LSTM to capture the sequential dependence of the historical motion state and obtain the sequential embedding representation, which enabled relational embedding in all directions through the self-attention mechanism to obtain the relational embedding representation. The sequential embedding representation and the relational embedding representation were concatenated and input into a prediction layer consisting of a fully connected neural network to generate nonlinear prediction components, which were added to the linear prediction components generated by the autoregressive module parallel to the above structure to generate the final prediction. The model was trained using a 'pre-training + fine-tuning' approach. In the validation experiments, 304 respiratory motion trajectories were used for model pre-training, and 7 evaluation samples were used for model testing.

RESULTS

The proposed prediction model achieved more accurate prediction results than other methods. For the 7 evaluation samples with different delay time, the proposed prediction model achieved a reduction of absolute deviations in the 3D directions by over 70%.

CONCLUSION

The proposed model is capable of accurate prediction of respiratory motion and can thus help to reduce system delay in precise radiotherapy.

Machine Learning Radiomics Model for External and Internal Respiratory Motion Correlation Prediction in Lung Tumor

Objectives: The complexity and specificity of lung tumor motion render it necessary to determine the external and internal correlation individually before applying indirect tumor tracking. However, the correlation cannot be determined from patient respiratory and tumor clinical characteristics before treatment. The purpose of this study is to present a machine learning model for an external/internal correlation prediction that is based on computed tomography (CT) radiomic features. Methods: 4-dimensional computed tomography (4DCT) images of 67 patients were collected retrospectively, and the external/internal correlation of lung tumor was calculated based on Spearman's rank correlation coefficient. Radiomic features were extracted from average intensity projection and the light gradient boosting machine (LightGBM)-based cross-validation (the recursive elimination method) was used for feature selection. The LightGBM framework forecasting models with classification thresholds 0.7, 0.8, and 0.9 are established using stratified 5-fold cross-validation. Model performance was assessed using receiver operating characteristics, sensitivity, and specificity. Results: There were 16, 18, and 13 features selected for models 0.7, 0.8, and 0.9, respectively. Texture features are of great importance in external/internal correlation prediction compared to other features in all models. The sensitivities of the predictions in models 0.7, 0.8, and 0.9 were 0.800 ± 0.126, 0.829 ± 0.140, and 0.864 ± 0.086, respectively. The specificities were 0.771 ± 0.114, 0.936 ± 0.0581, and 0.839 ± 0.101, whereas the area under the curve (AUC) was 0.837, 0.946, and 0.877, respectively. Conclusions: Our findings indicate that radiomics is an effective tool for respiratory motion correlation prediction, which can extract tumor motion characteristics. We proposed a machine learning framework for correlation prediction in the motion management strategy for lung tumor patients.

Artificial intelligence in radiotherapy

Radiotherapy is a discipline closely integrated with computer science. Artificial intelligence (AI) has developed rapidly over the past few years. With the explosive growth of medical big data, AI promises to revolutionize the field of radiotherapy through highly automated workflow, enhanced quality assurance, improved regional balances of expert experiences, and individualized treatment guided by multi-omics. In addition to independent researchers, the increasing number of large databases, biobanks, and open challenges significantly facilitated AI studies on radiation oncology. This article reviews the latest research, clinical applications, and challenges of AI in each part of radiotherapy including image processing, contouring, planning, quality assurance, motion management, and outcome prediction. By summarizing cutting-edge findings and challenges, we aim to inspire researchers to explore more future possibilities and accelerate the arrival of AI radiotherapy.

Unsupervised inter-frame motion correction for whole-body dynamic PET using convolutional long short-term memory in a convolutional neural network

Subject motion in whole-body dynamic PET introduces inter-frame mismatch and seriously impacts parametric imaging. Traditional non-rigid registration methods are generally computationally intense and time-consuming. Deep learning approaches are promising in achieving high accuracy with fast speed, but have yet been investigated with consideration for tracer distribution changes or in the whole-body scope. In this work, we developed an unsupervised automatic deep learning-based framework to correct inter-frame body motion. The motion estimation network is a convolutional neural network with a combined convolutional long short-term memory layer, fully utilizing dynamic temporal features and spatial information. Our dataset contains 27 subjects each under a 90-min FDG whole-body dynamic PET scan. Evaluating performance in motion simulation studies and a 9-fold cross-validation on the human subject dataset, compared with both traditional and deep learning baselines, we demonstrated that the proposed network achieved the lowest motion prediction error, obtained superior performance in enhanced qualitative and quantitative spatial alignment between parametric Ki and Vb images, and significantly reduced parametric fitting error. We also showed the potential of the proposed motion correction method for impacting downstream analysis of the estimated parametric images, improving the ability to distinguish malignant from benign hypermetabolic regions of interest. Once trained, the motion estimation inference time of our proposed network was around 460 times faster than the conventional registration baseline, showing its potential to be easily applied in clinical settings.

Real-Time 2D MR Cine From Beam Eye's View With Tumor-Volume Projection to Ensure Beam-to-Tumor Conformality for MR-Guided Radiotherapy of Lung Cancer

Purpose

To minimize computation latency using a predictive strategy to retrieve and project tumor volume onto 2D MR beam eye’s view (BEV) cine from time-resolved four-dimensional magnetic resonance imaging (TR-4DMRI) libraries (inhalation/exhalation) for personalized MR-guided intensity-modulated radiotherapy (IMRT) or volumetric-modulated arc therapy (VMAT).

Methods

Two time-series forecasting algorithms, autoregressive (AR) modeling and deep-learning-based long short-term memory (LSTM), were applied to predict the diaphragm position in the next 2D BEV cine to identify a motion-matched and hysteresis-accounted image to retrieve the tumor volume from the inhalation/exhalation TR-4DMRI libraries. Three 40-s TR-4DMRI (2 Hz, 3 × 80 images) per patient of eight lung cancer patients were used to create patient-specific inhalation/exhalation 4DMRI libraries, extract diaphragmatic waveforms, and interpolate them to f = 4 and 8 Hz to match 2D cine frame rates. Along a (40•f)-timepoint waveform, 30•f training timepoints were moved forward to produce 3×(10•f-1) predictions. The accuracy of position prediction was assessed against the waveform ground truth. The accuracy of tumor volume projections was evaluated using the center-of-mass difference (∆COM) and Dice similarity index against the TR-4DMRI ground truth for both IMRT (six beam angles, 30° interval) and VMAT (240/480 beam angles, 1.5°/0.75° interval, at 4/8 Hz, respectively).

Results

The accuracy of the first-timepoint prediction is 0.36 ± 0.10 mm (AR) and 0.62 ± 0.21 mm (LSTM) at 4 Hz and 0.06 ± 0.02 mm (AR) and 0.18 ± 0.06 mm (LSTM) at 8 Hz. A 10%–20% random error in prediction-library matching increases the overall uncertainty slightly. For both IMRT and VMAT, the accuracy of projected tumor volume contours on 2D BEV cine is ∆COM = 0.39 ± 0.13 mm and DICE = 0.97 ± 0.02 at 4 Hz and ∆COM = 0.10 ± 0.04 mm and DICE = 1.00 ± 0.00 at 8Hz.

Conclusion

This study demonstrates the feasibility of accurately predicting respiratory motion during 2D BEV cine imaging, identifying a motion-matched and hysteresis-accounted tumor volume, and projecting tumor volume contour on 2D BEV cine for real-time assessment of beam-to-tumor conformality, promising for optimal personalized MR-guided radiotherapy.

Prediction of the position of external markers using a recurrent neural network trained with unbiased online recurrent optimization for safe lung cancer radiotherapy

BACKGROUND AND OBJECTIVE

During lung cancer radiotherapy, the position of infrared reflective objects on the chest can be recorded to estimate the tumor location. However, radiotherapy systems have a latency inherent to robot control limitations that impedes the radiation delivery precision. Prediction with online learning of recurrent neural networks (RNN) allows for adaptation to non-stationary respiratory signals, but classical methods such as real-time recurrent learning (RTRL) and truncated backpropagation through time are respectively slow and biased. This study investigates the capabilities of unbiased online recurrent optimization (UORO) to forecast respiratory motion and enhance safety in lung radiotherapy.

METHODS

We used nine observation records of the three-dimensional (3D) position of three external markers on the chest and abdomen of healthy individuals breathing during intervals from 73s to 222s. The sampling frequency was 10Hz, and the amplitudes of the recorded trajectories range from 6mm to 40mm in the superior-inferior direction. We forecast the 3D location of each marker simultaneously with a horizon value (the time interval in advance for which the prediction is made) between 0.1s and 2.0s, using an RNN trained with UORO. We compare its performance with an RNN trained with RTRL, least mean squares (LMS), and offline linear regression. We provide closed-form expressions for quantities involved in the gradient loss calculation in UORO, thereby making its implementation efficient. Training and cross-validation were performed during the first minute of each sequence.

RESULTS

On average over the horizon values considered and the nine sequences, UORO achieves the lowest root-mean-square (RMS) error and maximum error among the compared algorithms. These errors are respectively equal to 1.3mm and 8.8mm, and the prediction time per time step was lower than 2.8ms (Dell Intel core i9-9900K 3.60 GHz). Linear regression has the lowest RMS error for the horizon values 0.1s and 0.2s, followed by LMS for horizon values between 0.3s and 0.5s, and UORO for horizon values greater than 0.6s.

CONCLUSIONS

UORO can accurately predict the 3D position of external markers for intermediate to high response times with an acceptable time performance. This will help limit unwanted damage to healthy tissues caused by radiotherapy.

Correlation of Optical Surface Respiratory Motion Signal and Internal Lung and Liver Tumor Motion: A Retrospective Single-Center Observational Study

Purpose: Surface-guided radiation therapy (SGRT) application has limitations. This study aimed to explore the relationship between patient characteristics and their external/internal correlation to qualitatively assess the external/internal correlation in a particular patient. Methods: Liver and lung cancer patients treated with radiotherapy in our institution were retrospectively analyzed. The external/internal correlation were calculated with Spearman correlation coefficient (SCC) and SCC after support vector regression (SVR) fitting (SCC_svr). The relationship between the external/internal correlation and magnitudes of motion of the tumor and external marker (A_i, A_e), tumor volume V_t, patient age, gender, and tumor location were explored. Results: The external/internal motions of liver and lung cancer patients were strongly correlated in the S-I direction, with mean SCC_svr values of 0.913 and 0.813. The correlation coefficients between the external/internal correlations and the patients’ characteristics (A_i, A_e, V_t, and age) were all smaller than 0.5; A_i, A_e and liver tumor volumes were positively correlated with the strength of the external/internal correlation, while lung tumor volumes and patient age were negative. The external/internal correlations in males and females were roughly equal, and the external/internal correlations in patients with peripheral lung cancers were stronger than those in patients with central lung cancers. Conclusion: The external/internal correlation shows great individual differences. The effects of A_i, A_e, V_t, and age are weakly to moderately correlated. Our results suggest the necessity of individualized assessment of patient's external/internal motion correlation prior to the application of SGRT technique for breath motion monitoring.

Current-Visit and Next-Visit Prediction for Fatty Liver Disease With a Large-Scale Dataset: Model Development and Performance Comparison

BACKGROUND

Fatty liver disease (FLD) arises from the accumulation of fat in the liver and may cause liver inflammation, which, if not well controlled, may develop into liver fibrosis, cirrhosis, or even hepatocellular carcinoma.

OBJECTIVE

We describe the construction of machine-learning models for current-visit prediction (CVP), which can help physicians obtain more information for accurate diagnosis, and next-visit prediction (NVP), which can help physicians provide potential high-risk patients with advice to effectively prevent FLD.

METHODS

The large-scale and high-dimensional dataset used in this study comes from Taipei MJ Health Research Foundation in Taiwan. We used one-pass ranking and sequential forward selection (SFS) for feature selection in FLD prediction. For CVP, we explored multiple models, including k-nearest-neighbor classifier (KNNC), Adaboost, support vector machine (SVM), logistic regression (LR), random forest (RF), Gaussian naïve Bayes (GNB), decision trees C4.5 (C4.5), and classification and regression trees (CART). For NVP, we used long short-term memory (LSTM) and several of its variants as sequence classifiers that use various input sets for prediction. Model performance was evaluated based on two criteria: the accuracy of the test set and the intersection over union/coverage between the features selected by one-pass ranking/SFS and by domain experts. The accuracy, precision, recall, F-measure, and area under the receiver operating characteristic curve were calculated for both CVP and NVP for males and females, respectively.

RESULTS

After data cleaning, the dataset included 34,856 and 31,394 unique visits respectively for males and females for the period 2009-2016. The test accuracy of CVP using KNNC, Adaboost, SVM, LR, RF, GNB, C4.5, and CART was respectively 84.28%, 83.84%, 82.22%, 82.21%, 76.03%, 75.78%, and 75.53%. The test accuracy of NVP using LSTM, bidirectional LSTM (biLSTM), Stack-LSTM, Stack-biLSTM, and Attention-LSTM was respectively 76.54%, 76.66%, 77.23%, 76.84%, and 77.31% for fixed-interval features, and was 79.29%, 79.12%, 79.32%, 79.29%, and 78.36%, respectively, for variable-interval features.

CONCLUSIONS

This study explored a large-scale FLD dataset with high dimensionality. We developed FLD prediction models for CVP and NVP. We also implemented efficient feature selection schemes for current- and next-visit prediction to compare the automatically selected features with expert-selected features. In particular, NVP emerged as more valuable from the viewpoint of preventive medicine. For NVP, we propose use of feature set 2 (with variable intervals), which is more compact and flexible. We have also tested several variants of LSTM in combination with two feature sets to identify the best match for male and female FLD prediction. More specifically, the best model for males was Stack-LSTM using feature set 2 (with 79.32% accuracy), whereas the best model for females was LSTM using feature set 1 (with 81.90% accuracy).

A review of artificial intelligence applications for motion tracking in radiotherapy

During radiotherapy, the organs and tumour move as a result of the dynamic nature of the body; this is known as intrafraction motion. Intrafraction motion can result in tumour underdose and healthy tissue overdose, thereby reducing the effectiveness of the treatment while increasing toxicity to the patients. There is a growing appreciation of intrafraction target motion management by the radiation oncology community. Real-time image-guided radiation therapy (IGRT) can track the target and account for the motion, improving the radiation dose to the tumour and reducing the dose to healthy tissue. Recently, artificial intelligence (AI)-based approaches have been applied to motion management and have shown great potential. In this review, four main categories of motion management using AI are summarised: marker-based tracking, markerless tracking, full anatomy monitoring and motion prediction. Marker-based and markerless tracking approaches focus on tracking the individual target throughout the treatment. Full anatomy algorithms monitor for intrafraction changes in the full anatomy within the field of view. Motion prediction algorithms can be used to account for the latencies due to the time for the system to localise, process and act.