Physically constrained voxel-based penalty adaptation for ultra-fast IMRT planning

Deep reinforcement learning in radiation therapy planning optimization: A comprehensive review

PURPOSE

The formulation and optimization of radiation therapy plans are complex and time-consuming processes that heavily rely on the expertise of medical physicists. Consequently, there is an urgent need for automated optimization methods. Recent advancements in reinforcement learning, particularly deep reinforcement learning (DRL), show great promise for automating radiotherapy planning. This review summarizes the current state of DRL applications in this field, evaluates their effectiveness, and identifies challenges and future directions.

METHODS

A systematic search was conducted in Google Scholar, PubMed, IEEE Xplore, and Scopus using keywords such as "deep reinforcement learning", "radiation therapy", and "treatment planning". The extracted data were synthesized for an overview and critical analysis.

RESULTS

The application of deep reinforcement learning in radiation therapy plan optimization can generally be divided into three categories: optimizing treatment planning parameters, directly optimizing machine parameters, and adaptive radiotherapy. From the perspective of disease sites, DRL has been applied to cervical cancer, prostate cancer, vestibular schwannoma, and lung cancer. Regarding types of radiation therapy, it has been used in HDRBT, IMRT, SBRT, VMAT, GK, and Cyberknife.

CONCLUSIONS

Deep reinforcement learning technology has played a significant role in advancing the automated optimization of radiation therapy plans. However, there is still a considerable gap before it can be widely applied in clinical settings due to three main reasons: inefficiency, limited methods for quality assessment, and poor interpretability. To address these challenges, significant research opportunities exist in the future, such as constructing evaluators, parallelized training, and exploring continuous action spaces.

Implementation and evaluation of an intelligent automatic treatment planning robot for prostate cancer stereotactic body radiation therapy

PURPOSE

We previously developed a virtual treatment planner (VTP), an artificial intelligence robot, operating a treatment planning system (TPS). Using deep reinforcement learning guided by human knowledge, we trained the VTP to autonomously adjust relevant parameters in treatment plan optimization, similar to a human planner, to generate high-quality plans for prostate cancer stereotactic body radiation therapy (SBRT). This study describes the clinical implementation and evaluation of VTP.

MATERIALS AND METHODS

We integrate VTP with Eclipse TPS using scripting Application Programming Interface. VTP observes dose-volume histograms of relevant structures, decides how to adjust dosimetric constraints, including doses, volumes, and weighting factors, and applies the adjustments to the TPS interface to launch the optimization engine. This process continues until a high-quality plan is achieved. We evaluated VTP's performance using the prostate SBRT case from the 2016 American Association of Medical Dosimetrist/Radiosurgery Society plan study with its plan scoring system, and compared to human-generated plans submitted to the challenge. Using the same scoring system, we also compared the plan quality of 36 prostate SBRT cases (20 planned with IMRT and 16 planned with VMAT) treated at our institution for both VTP and human-generated plans.

RESULTS

In the plan study case, VTP achieved a score of 142.1/150.0, ranking the third in the competition (median 134.6). For the clinical cases, VTP achieved 110.6 ± 6.5 for 20 IMRT plans and 126.2 ± 4.7 for 16 VMAT plans, similar to scores of human-generated plans with 110.4 ± 7.0 for IMRT plans and 125.4 ± 4.4 for VMAT plans. The workflow, plan quality and planning time of VTP were reviewed to be satisfactory by experienced physicists.

CONCLUSION

We successfully implemented VTP to operate a TPS for autonomous human-like treatment planning for prostate SBRT.

A conjugate gradient-assisted multi-objective evolutionary algorithm for fluence map optimization in radiotherapy treatment

Intensity-modulated radiotherapy (IMRT) is one of the most applied techniques for cancer radiotherapy treatment. The fluence map optimization is an essential part of IMRT plan designing, which has a significant impact on the radiotherapy treatment effect. In fact, the treatment planing of IMRT is an inverse multi-objective optimization problem. Existing approaches of solving the fluence map optimization problem (FMOP) obtain a satisfied treatment plan via trying different coupling weights, the optimization process needs to be conducted many times and the coupling weight setting is completely based on the experience of a radiation physicist. For fast obtaining diverse high-quality radiotherapy plans, this paper formulates the FMOP into a three-objective optimization problem, and proposes a conjugate gradient-assisted multi-objective evolutionary algorithm (CG-MOEA) to solve it. The proposed algorithm does not need to set the coupling weights and can produce the diverse radiotherapy plans within a single run. Moreover, the convergence speed is further accelerated by an adaptive local search strategy based on the conjugate-gradient method. Compared with five state-of-the-art multi-objective evolutionary algorithms (MOEAs), the proposed CG-MOEA can obtain the best hypervolume (HV) values and dose–volume histogram (DVH) performance on five clinical cases in cancer radiotherapy. Moreover, the proposed algorithm not only obtains the more optimal solution than traditional method used to solve the FMOP, but also can find diverse Pareto solution set which can be provided to radiation physicist to select the best treatment plan. The proposed algorithm outperforms dose-volume histogram state-of-the-art multi-objective evolutionary algorithms and traditional method for FMOP on five clinical cases in cancer radiotherapy.

Automatic Inverse Treatment Planning of Gamma Knife Radiosurgery via Deep Reinforcement Learning

PURPOSE

Several inverse planning algorithms have been developed for Gamma Knife (GK) radiosurgery to determine a large number of plan parameters via solving an optimization problem, which typically consists of multiple objectives. The priorities among these objectives need to be repetitively adjusted to achieve a clinically good plan for each patient. This study aimed to achieve automatic and intelligent priority-tuning, by developing a deep reinforcement learning (DRL) based method to model the tuning behaviors of human planners.

METHODS

We built a priority-tuning policy network using deep convolutional neural networks. Its input was a vector composed of multiple plan metrics that were used in our institution for GK plan evaluation. The network can determine which tuning action to take, based on the observed quality of the intermediate plan. We trained the network using an end-to-end DRL framework to approximate the optimal action-value function. A scoring function was designed to measure the plan quality to calculate the received reward of a tuning action.

RESULTS

Vestibular schwannoma was chosen as the test bed in this study. The number of training, validation and testing cases were 5, 5, and 16, respectively. For these three datasets, the average scores of the initial plans obtained with a same initial priority set were 3.63 ± 1.34, 3.83 ± 0.86 and 4.20 ± 0.78, respectively, while can be improved to 5.28 ± 0.23, 4.97 ± 0.44 and 5.22 ± 0.26 through manual priority tuning by human expert planners. Our network achieved competitive results with 5.42 ± 0.11, 5.10 ± 0. 42, 5.28 ± 0.20, respectively.

CONCLUSIONS

Our network can generate GK plans of comparable or slightly higher quality comparing with the plans generated by human planners via manual priority tuning for vestibular schwannoma cases. The network can potentially be incorporated into the clinical workflow as a planning assistance to improve GK planning efficiency and help to reduce plan quality variation caused by inter-planner variability. We also hope that our method can reduce the workload of GK planners and allow them to spend more time on more challenging cases. This article is protected by copyright. All rights reserved.

A hierarchical deep reinforcement learning framework for intelligent automatic treatment planning of prostate cancer intensity modulated radiation therapy

Purpose.We have previously proposed an intelligent automatic treatment planning (IATP) framework that builds a virtual treatment planner network (VTPN) to operate a treatment planning system (TPS) to generate high-quality radiation therapy (RT) treatment plans. While the potential of IATP in automating RT treatment planning has been demonstrated, its poor scalability caused by an almost linear growth of network size with the number of treatment planning parameters (TPPs) is a bottleneck, preventing its application in complicate, but clinically relevant treatment planning problems. The decision-making behavior of the trained network is hard to understand. Motivated by the decision-making process of a human planner, this study proposes a hierarchical IATP framework.Methods and materials.The hierarchical VTPN (HieVTPN) consists of three networks, i.e. Structure-Net, Parameter-Net, and Action-Net. When interacting with a TPS, the networks are employed in a sequential order in each step to decide the structure to adjust, the TPP to adjust for the selected structure, and the specific adjustment manner for the parameter, respectively. We developed an end-to-end hierarchical deep reinforcement learning scheme to simultaneously train the three networks. We then evaluated the effectiveness of the proposed framework in the treatment planning problems for prostate cancer intensity modulated RT (IMRT) and stereotactic body RT (SBRT). We benchmarked the performance of our approach by comparing plans made by VTPN of a parallel architecture, and the human plans submitted for competition in the 2016 American Association of Medical Dosimetrist (AAMD)/Radiosurgery Society (RSS) Plan Study. We analyzed scalability of the network size with respect to the number of TPPs. Numerical experiments were also performed to understand the rationale of the decision-making behaviors of the trained HieVTPN.Results.Both HieVTPNs for prostate IMRT and SBRT were trained successfully using 10 training patient cases and 5 validation cases. For IMRT, HieVTPN was able to generate high-quality plans for 59 testing patient cases that were not included in training process, achieving an average plan score of 8.62 (±0.83), with 9 being the maximal score. The score was comparable to that of the VTPN, 8.45 (±0.48). For SBRT planning, HieVTPN achieved an average plan score of 139.07 on five testing patient cases compared to the score of 132.21 averaged over the human plans summited for competition in AAMD/RSS plan study. Different from VTPN with network size linearly scaling with the number of TPPs, the network size of HieVTPN is almost independent of the number of TPPs. It was also observed that the decision-making behaviors of HieVTPN were understandable and generally agreed with the human experience.Conclusions.With the scalability and explainability, the hierarchical IATP framework is more favorable than the previous framework in terms of handling treatment planning problems involving a large number of TPPs.

Improving efficiency of training a virtual treatment planner network via knowledge-guided deep reinforcement learning for intelligent automatic treatment planning of radiotherapy

PURPOSE

We previously proposed an intelligent automatic treatment planning framework for radiotherapy, in which a virtual treatment planner network (VTPN) is built using deep reinforcement learning (DRL) to operate a treatment planning system (TPS) by adjusting treatment planning parameters in it to generate high-quality plans. We demonstrated the potential feasibility of this idea in prostate cancer intensity-modulated radiation therapy (IMRT). Despite the success, the process to train a VTPN via the standard DRL approach with an ϵ-greedy algorithm was time-consuming. The required training time was expected to grow with the complexity of the treatment planning problem, preventing the development of VTPN for more complicated but clinically relevant scenarios. In this study, we proposed a novel knowledge-guided DRL (KgDRL) approach that incorporated knowledge from human planners to guide the training process to improve the efficiency of training a VTPN.

METHOD

Using prostate cancer IMRT as a test bed, we first summarized a number of rules in the actions of adjusting treatment planning parameters of our in-house TPS. During the training process of VTPN, in addition to randomly navigating the large state-action space, as in the standard DRL approach using the ϵ-greedy algorithm, we also sampled actions defined by the rules. The priority of sampling actions from rules decreased over the training process to encourage VTPN to explore new policy on parameter adjustment that were not covered by the rules. To test this idea, we trained a VTPN using KgDRL and compared its performance with another VTPN trained using the standard DRL approach. Both networks were trained using 10 training patient cases and five additional cases for validation, while another 59 cases were employed for the evaluation purpose.

RESULTS

It was found that both VTPNs trained via KgDRL and standard DRL spontaneously learned how to operate the in-house TPS to generate high-quality plans, achieving plan quality scores of 8.82 (±0.29) and 8.43 (±0.48), respectively. Both VTPNs outperformed treatment planning purely based on the rules, which had a plan score of 7.81 (±1.59). VTPN trained with eight episodes using KgDRL was able to perform similar to that trained using DRL with 100 epochs. The training time was reduced from more than a week to ~13 hrs.

CONCLUSION

The proposed KgDRL framework was effective in accelerating the training process of a VTPN by incorporating human knowledge, which will facilitate the development of VTPN for more complicated treatment planning scenarios.

Operating a treatment planning system using a deep-reinforcement learning-based virtual treatment planner for prostate cancer intensity-modulated radiation therapy treatment planning

PURPOSE

In the treatment planning process of intensity-modulated radiation therapy (IMRT), a human planner operates the treatment planning system (TPS) to adjust treatment planning parameters, for example, dose volume histogram (DVH) constraints' locations and weights, to achieve a satisfactory plan for each patient. This process is usually time-consuming, and the plan quality depends on planer's experience and available planning time. In this study, we proposed to model the behaviors of human planners in treatment planning by a deep reinforcement learning (DRL)-based virtual treatment planner network (VTPN), such that it can operate the TPS in a human-like manner for treatment planning.

METHODS AND MATERIALS

Using prostate cancer IMRT as an example, we established the VTPN using a deep neural network developed. We considered an in-house optimization engine with a weighted quadratic objective function. Virtual treatment planner network was designed to observe an intermediate plan DVHs and decide the action to improve the plan by changing weights and threshold dose in the objective function. We trained the VTPN in an end-to-end DRL process in 10 patient cases. A plan score was used to measure plan quality. We demonstrated the feasibility and effectiveness of the trained VTPN in another 64 patient cases.

RESULTS

Virtual treatment planner network was trained to spontaneously learn how to adjust treatment planning parameters to generate high-quality treatment plans. In the 64 testing cases, with initialized parameters, quality score was 4.97 (±2.02), with 9.0 being the highest possible score. Using VTPN to perform treatment planning improved quality score to 8.44 (±0.48).

CONCLUSIONS

To our knowledge, this was the first time that intelligent treatment planning behaviors of human planner in external beam IMRT are autonomously encoded in an artificial intelligence system. The trained VTPN is capable of behaving in a human-like way to produce high-quality plans.

Intelligent inverse treatment planning via deep reinforcement learning, a proof-of-principle study in high dose-rate brachytherapy for cervical cancer

Inverse treatment planning in radiation therapy is formulated as solving optimization problems. The objective function and constraints consist of multiple terms designed for different clinical and practical considerations. Weighting factors of these terms are needed to define the optimization problem. While a treatment planning optimization engine can solve the optimization problem with given weights, adjusting the weights to yield a high-quality plan is typically performed by a human planner. Yet the weight-tuning task is labor intensive, time consuming, and it critically affects the final plan quality. An automatic weight-tuning approach is strongly desired. The procedure of weight adjustment to improve the plan quality is essentially a decision-making problem. Motivated by the tremendous success in deep learning for decision making with human-level intelligence, we propose a novel framework to adjust the weights in a human-like manner. This study used inverse treatment planning in high-dose-rate brachytherapy (HDRBT) for cervical cancer as an example. We developed a weight-tuning policy network (WTPN) that observes dose volume histograms of a plan and outputs an action to adjust organ weighting factors, similar to the behaviors of a human planner. We trained the WTPN via end-to-end deep reinforcement learning. Experience replay was performed with the epsilon greedy algorithm. After training was completed, we applied the trained WTPN to guide treatment planning of five testing patient cases. It was found that the trained WTPN successfully learnt the treatment planning goals and was able to guide the weight tuning process. On average, the quality score of plans generated under the WTPN's guidance was improved by ~8.5% compared to the initial plan with arbitrarily set weights, and by 10.7% compared to the plans generated by human planners. To our knowledge, this was the first time that a tool was developed to adjust organ weights for the treatment planning optimization problem in a human-like fashion based on intelligence learnt from a training process, which was different from existing strategies based on pre-defined rules. The study demonstrated potential feasibility to develop intelligent treatment planning approaches via deep reinforcement learning.

Voxel-based automatic multi-criteria optimization for intensity modulated radiation therapy

Background

Automatic multi-criteria optimization is necessary for intensity modulated radiation therapy (IMRT) because of low planning efficiency and large plan quality uncertainty in current clinical practice. Most studies focused on imitating dosimetrists’ planning procedures to automate this process and ignored the fact that organ-based objective functions typically used in commercial treatment planning systems (such as dose-volume function) usually lead to sub-optimal plans. To guarantee the optimum results and to automate this process, we incorporate an improved automation strategy and a voxel-based optimization algorithm to generate a novel automatic multi-criteria optimization framework. We then evaluate it in clinical cases.

Methods

This novel automatic multi-criteria optimization framework incorporates a ranked priority-list based automatic constraints adjustment strategy and an in-house developed voxel-based optimization algorithm. Constraints are sequentially adjusted following a pre-defined priority list. Afterward, a voxel-based fluence map optimization (FMO) with an orientation to the newly updated constraints is launched to find a Pareto optimal solution. Loops of constraints adjustment are repeated until each of them could not be relaxed or tightened. The feasibility of the framework is evaluated with 10 automatic generated gynecology (GYN) cancer IMRT cases by comparing the dosimetric performance with the original.

Results

Plan quality improvement is observed for our automatic multi-criteria optimization method. Comparable DVHs are found for the planning target volume (PTV), but with better organs-at-risk (OAR) dose sparing. Among 13 evaluated dosimetric endpoints, 5 of them show significant improvements in automatically generated plans compared with the original plans. Investigation of improvement tendency during optimization exhibits gradual change as the optimization stage proceeds. An initial voxel-based optimization stage and in-low-priority dosimetric criteria tighten can significantly contribute to the optimization procedure.

Conclusions

We have successfully developed an automatic multi-criteria optimization framework that can dramatically reduce the current trial-and-error patterned planning workload while affording an efficient method to assure high plan quality consistency. This optimization framework is expected to greatly facilitate precise radiation therapy because of its advantages of planning efficiency and plan quality improvement.

[Constraint priority list-based multi-objective optimization for intensity-modulated radiation therapy]

In intensity-modulated radiation therapy (IMRT), it is time-consuming to repeatedly adjust the objectives manually to obtain the best tradeoff between the prescribed dose of the planning target volume and sparing the organs-at-risk. Here we propose a new method to realize automatic multi-objective IMRT optimization, which quantifies the clinical preferences into the constraint priority list and adjusts the dose constraints based on the list to obtain the optimal solutions under the dose constraints. This method contains automatic adjustment mechanism of the dose constraint and automatic voxel weighting factor-based FMO model. Every time the dose constraint is adjusted, the voxel weighting factor-based FMO model is launched to find a global optimal solution that satisfied the current constraints. We tested the feasibility and effectiveness of this method in 6 cases of cervical cancer with IMRT by comparing the original plan and the automatic optimization plan generated by this method. The results showed that with the same PTV coverage and uniformity, the automatic optimization plan had a better a dose sparing of the organs-at-risk and a better plan quality than the original plan, and resulted in obvious reductions of the average V45 of the rectum from (41.99∓13.31)% to (32.55∓22.27)% and of the bladder from (44.37∓4.08)% to (28.99∓15.25)%.

Feasibility of online IMPT adaptation using fast, automatic and robust dose restoration

Intensity-modulated proton therapy (IMPT) offers excellent dose conformity and healthy tissue sparing, but it can be substantially compromised in the presence of anatomical changes. A major dosimetric effect is caused by density changes, which alter the planned proton range in the patient. Three different methods, which automatically restore an IMPT plan dose on a daily CT image were implemented and compared: (1) simple dose restoration (DR) using optimization objectives of the initial plan, (2) voxel-wise dose restoration (vDR), and (3) isodose volume dose restoration (iDR). Dose restorations were calculated for three different clinical cases, selected to test different capabilities of the restoration methods: large range adaptation, complex dose distributions and robust re-optimization. All dose restorations were obtained in less than 5 min, without manual adjustments of the optimization settings. The evaluation of initial plans on repeated CTs showed large dose distortions, which were substantially reduced after restoration. In general, all dose restoration methods improved DVH-based scores in propagated target volumes and OARs. Analysis of local dose differences showed that, although all dose restorations performed similarly in high dose regions, iDR restored the initial dose with higher precision and accuracy in the whole patient anatomy. Median dose errors decreased from 13.55 Gy in distorted plan to 9.75 Gy (vDR), 6.2 Gy (DR) and 4.3 Gy (iDR). High quality dose restoration is essential to minimize or eventually by-pass the physician approval of the restored plan, as long as dose stability can be assumed. Motion (as well as setup and range uncertainties) can be taken into account by including robust optimization in the dose restoration. Restoring clinically-approved dose distribution on repeated CTs does not require new ROI segmentation and is compatible with an online adaptive workflow.