1
|
Babier A, Mahmood R, Zhang B, Alves VGL, Barragán-Montero AM, Beaudry J, Cardenas CE, Chang Y, Chen Z, Chun J, Diaz K, Eraso HD, Faustmann E, Gaj S, Gay S, Gronberg M, Guo B, He J, Heilemann G, Hira S, Huang Y, Ji F, Jiang D, Giraldo JCJ, Lee H, Lian J, Liu S, Liu KC, Marrugo J, Miki K, Nakamura K, Netherton T, Nguyen D, Nourzadeh H, Osman AFI, Peng Z, Muñoz JDQ, Ramsl C, Rhee DJ, Rodriguez JD, Shan H, Siebers JV, Soomro MH, Sun K, Hoyos AU, Valderrama C, Verbeek R, Wang E, Willems S, Wu Q, Xu X, Yang S, Yuan L, Zhu S, Zimmermann L, Moore KL, Purdie TG, McNiven AL, Chan TCY. OpenKBP-Opt: an international and reproducible evaluation of 76 knowledge-based planning pipelines. Phys Med Biol 2022; 67:10.1088/1361-6560/ac8044. [PMID: 36093921 PMCID: PMC10696540 DOI: 10.1088/1361-6560/ac8044] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/26/2022] [Accepted: 07/11/2022] [Indexed: 11/12/2022]
Abstract
Objective.To establish an open framework for developing plan optimization models for knowledge-based planning (KBP).Approach.Our framework includes radiotherapy treatment data (i.e. reference plans) for 100 patients with head-and-neck cancer who were treated with intensity-modulated radiotherapy. That data also includes high-quality dose predictions from 19 KBP models that were developed by different research groups using out-of-sample data during the OpenKBP Grand Challenge. The dose predictions were input to four fluence-based dose mimicking models to form 76 unique KBP pipelines that generated 7600 plans (76 pipelines × 100 patients). The predictions and KBP-generated plans were compared to the reference plans via: the dose score, which is the average mean absolute voxel-by-voxel difference in dose; the deviation in dose-volume histogram (DVH) points; and the frequency of clinical planning criteria satisfaction. We also performed a theoretical investigation to justify our dose mimicking models.Main results.The range in rank order correlation of the dose score between predictions and their KBP pipelines was 0.50-0.62, which indicates that the quality of the predictions was generally positively correlated with the quality of the plans. Additionally, compared to the input predictions, the KBP-generated plans performed significantly better (P< 0.05; one-sided Wilcoxon test) on 18 of 23 DVH points. Similarly, each optimization model generated plans that satisfied a higher percentage of criteria than the reference plans, which satisfied 3.5% more criteria than the set of all dose predictions. Lastly, our theoretical investigation demonstrated that the dose mimicking models generated plans that are also optimal for an inverse planning model.Significance.This was the largest international effort to date for evaluating the combination of KBP prediction and optimization models. We found that the best performing models significantly outperformed the reference dose and dose predictions. In the interest of reproducibility, our data and code is freely available.
Collapse
Affiliation(s)
- Aaron Babier
- Department of Mechanical and Industrial Engineering, University of Toronto, Toronto, ON, Canada
- Vector Institute, Toronto, ON, Canada
| | - Rafid Mahmood
- Department of Mechanical and Industrial Engineering, University of Toronto, Toronto, ON, Canada
| | - Binghao Zhang
- Department of Mechanical and Industrial Engineering, University of Toronto, Toronto, ON, Canada
| | - Victor G L Alves
- Department of Radiation Oncology, University of Virginia Health System, Charlottesville, VA, United States of America
| | | | - Joel Beaudry
- Department of Radiation Oncology, Memorial Sloan Kettering Cancer Center, New York, NY, United States of America
| | - Carlos E Cardenas
- Department of Radiation Oncology, The University of Alabama at Birmingham, Birmingham, AL, United States of America
| | - Yankui Chang
- Department of Engineering and Applied Physics, University of Science and Technology of China, Hefei, People’s Republic of China
| | - Zijie Chen
- Shenying Medical Technology Co., Ltd., Shenzhen, Guangdong, People’s Republic of China
| | - Jaehee Chun
- Department of Radiation Oncology, Yonsei University College of Medicine, Seoul, Republic of Korea
| | - Kelly Diaz
- Department of Physics, National University of Colombia, Medellín, Colombia
| | - Harold David Eraso
- Department of Physics, National University of Colombia, Medellín, Colombia
| | - Erik Faustmann
- Atominstitut, Vienna University of Technology, Vienna, Austria
| | - Sibaji Gaj
- Department of Biomedical Engineering, Cleveland Clinic, Cleveland, OH, United States of America
| | - Skylar Gay
- Department of Radiation Physics, The University of Texas MD Anderson Cancer Center, Houston, TX, United States of America
| | - Mary Gronberg
- Department of Radiation Physics, The University of Texas MD Anderson Cancer Center, Houston, TX, United States of America
| | - Bingqi Guo
- Department of Radiation Oncology, Cleveland Clinic, Cleveland, OH, United States of America
| | - Junjun He
- Department of Biomedical Engineering, Shanghai Jiao Tong University, Shanghai, People’s Republic of China
| | - Gerd Heilemann
- Department of Radiation Oncology, Medical University of Vienna, Vienna, Austria
| | - Sanchit Hira
- Department of Biomedical Engineering, Johns Hopkins University, Baltimore, MD, United States of America
| | - Yuliang Huang
- Department of Radiation Oncology, Peking University Cancer Hospital and Institute, Beijing, People’s Republic of China
| | - Fuxin Ji
- Department of Electrical Engineering and Automation, Anhui University, Hefei, People’s Republic of China
| | - Dashan Jiang
- Department of Electrical Engineering and Automation, Anhui University, Hefei, People’s Republic of China
| | | | - Hoyeon Lee
- Department of Radiation Oncology, Massachusetts General Hospital, Boston, MA, United States of America
| | - Jun Lian
- Department of Radiation Oncology, University of North Carolina at Chapel Hill, Chapel Hill, NC, United States of America
| | - Shuolin Liu
- Department of Electrical Engineering and Automation, Anhui University, Hefei, People’s Republic of China
| | - Keng-Chi Liu
- Department of Medical Imaging, Taiwan AI Labs, Taipei, Taiwan
| | - José Marrugo
- Department of Physics, National University of Colombia, Medellín, Colombia
| | - Kentaro Miki
- Department Of Biomedical and Health Sciences, Hiroshima University, Hiroshima, Japan
| | - Kunio Nakamura
- Department of Biomedical Engineering, Cleveland Clinic, Cleveland, OH, United States of America
| | - Tucker Netherton
- Department of Radiation Physics, The University of Texas MD Anderson Cancer Center, Houston, TX, United States of America
| | - Dan Nguyen
- Medical Artificial Intelligence and Automation (MAIA) Laboratory, Department of Radiation Oncology, The University of Texas Southwestern Medical Center, Dallas, TX, United States of America
| | - Hamidreza Nourzadeh
- Department of Radiation Oncology, Thomas Jefferson University, Philadelphia, PA, United States of America
| | | | - Zhao Peng
- Department of Engineering and Applied Physics, University of Science and Technology of China, Hefei, People’s Republic of China
| | | | - Christian Ramsl
- Atominstitut, Vienna University of Technology, Vienna, Austria
| | - Dong Joo Rhee
- Department of Radiation Physics, The University of Texas MD Anderson Cancer Center, Houston, TX, United States of America
| | | | - Hongming Shan
- Institute of Science and Technology for Brain-inspired Intelligence, Fudan University, Shanghai, People’s Republic of China
| | - Jeffrey V Siebers
- Department of Radiation Oncology, University of Virginia Health System, Charlottesville, VA, United States of America
| | - Mumtaz H Soomro
- Department of Radiation Oncology, University of Virginia Health System, Charlottesville, VA, United States of America
| | - Kay Sun
- Studio Vodels, Atlanta, GA, United States of America
| | - Andrés Usuga Hoyos
- Department of Physics, National University of Colombia, Medellín, Colombia
| | - Carlos Valderrama
- Department of Physics, National University of Colombia, Medellín, Colombia
| | - Rob Verbeek
- Department Computer Science, Aalto University, Espoo, Finland
| | - Enpei Wang
- Shenying Medical Technology Co., Ltd., Shenzhen, Guangdong, People’s Republic of China
| | - Siri Willems
- Department of Electrical Engineering, KULeuven, Leuven, Belgium
| | - Qi Wu
- Department of Electrical Engineering and Automation, Anhui University, Hefei, People’s Republic of China
| | - Xuanang Xu
- Department of Biomedical Engineering, Rensselaer Polytechnic Institute, Troy, NY, United States of America
| | - Sen Yang
- Tencent AI Lab, Shenzhen, Guangdong, People’s Republic of China
| | - Lulin Yuan
- Department of Radiation Oncology, Virginia Commonwealth University Medical Center, Richmond, VA, United States of America
| | - Simeng Zhu
- Department of Radiation Oncology, Henry Ford Health System, Detroit, MI, United States of America
| | - Lukas Zimmermann
- Faculty of Health, University of Applied Sciences Wiener Neustadt, Wiener Neustadt, Austria
- Competence Center for Preclinical Imaging and Biomedical Engineering, University of Applied Sciences Wiener Neustadt, Wiener Neustadt, Austria
| | - Kevin L Moore
- Department of Radiation Oncology, University of California, San Diego, La Jolla, CA, United States of America
| | - Thomas G Purdie
- Radiation Medicine Program, UHN Princess Margaret Cancer Centre, Toronto, ON, Canada
- Department of Radiation Oncology, University of Toronto, Toronto, ON, Canada
- Techna Institute for the Advancement of Technology for Health, Toronto, ON, Canada
- Department of Medical Biophysics, University of Toronto, Toronto, ON, Canada
| | - Andrea L McNiven
- Radiation Medicine Program, UHN Princess Margaret Cancer Centre, Toronto, ON, Canada
- Department of Radiation Oncology, University of Toronto, Toronto, ON, Canada
| | - Timothy C Y Chan
- Department of Mechanical and Industrial Engineering, University of Toronto, Toronto, ON, Canada
- Vector Institute, Toronto, ON, Canada
- Techna Institute for the Advancement of Technology for Health, Toronto, ON, Canada
| |
Collapse
|