1
|
Rong Y, Chen Q, Fu Y, Yang X, Al-Hallaq HA, Wu QJ, Yuan L, Xiao Y, Cai B, Latifi K, Benedict SH, Buchsbaum JC, Qi XS. NRG Oncology Assessment of Artificial Intelligence Deep Learning-Based Auto-segmentation for Radiation Therapy: Current Developments, Clinical Considerations, and Future Directions. Int J Radiat Oncol Biol Phys 2024; 119:261-280. [PMID: 37972715 PMCID: PMC11023777 DOI: 10.1016/j.ijrobp.2023.10.033] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/08/2023] [Revised: 09/16/2023] [Accepted: 10/14/2023] [Indexed: 11/19/2023]
Abstract
Deep learning neural networks (DLNN) in Artificial intelligence (AI) have been extensively explored for automatic segmentation in radiotherapy (RT). In contrast to traditional model-based methods, data-driven AI-based models for auto-segmentation have shown high accuracy in early studies in research settings and controlled environment (single institution). Vendor-provided commercial AI models are made available as part of the integrated treatment planning system (TPS) or as a stand-alone tool that provides streamlined workflow interacting with the main TPS. These commercial tools have drawn clinics' attention thanks to their significant benefit in reducing the workload from manual contouring and shortening the duration of treatment planning. However, challenges occur when applying these commercial AI-based segmentation models to diverse clinical scenarios, particularly in uncontrolled environments. Contouring nomenclature and guideline standardization has been the main task undertaken by the NRG Oncology. AI auto-segmentation holds the potential clinical trial participants to reduce interobserver variations, nomenclature non-compliance, and contouring guideline deviations. Meanwhile, trial reviewers could use AI tools to verify contour accuracy and compliance of those submitted datasets. In recognizing the growing clinical utilization and potential of these commercial AI auto-segmentation tools, NRG Oncology has formed a working group to evaluate the clinical utilization and potential of commercial AI auto-segmentation tools. The group will assess in-house and commercially available AI models, evaluation metrics, clinical challenges, and limitations, as well as future developments in addressing these challenges. General recommendations are made in terms of the implementation of these commercial AI models, as well as precautions in recognizing the challenges and limitations.
Collapse
Affiliation(s)
- Yi Rong
- Mayo Clinic Arizona, Phoenix, AZ
| | - Quan Chen
- City of Hope Comprehensive Cancer Center Duarte, CA
| | - Yabo Fu
- Memorial Sloan Kettering Cancer Center, Commack, NY
| | | | | | | | - Lulin Yuan
- Virginia Commonwealth University, Richmond, VA
| | - Ying Xiao
- University of Pennsylvania/Abramson Cancer Center, Philadelphia, PA
| | - Bin Cai
- The University of Texas Southwestern Medical Center, Dallas, TX
| | | | - Stanley H Benedict
- University of California Davis Comprehensive Cancer Center, Sacramento, CA
| | | | - X Sharon Qi
- University of California Los Angeles, Los Angeles, CA
| |
Collapse
|
2
|
Luan S, Ou-Yang J, Yang X, Wei W, Xue X, Zhu B. A multi-modal vision-language pipeline strategy for contour quality assurance and adaptive optimization. Phys Med Biol 2024; 69:065005. [PMID: 38373347 DOI: 10.1088/1361-6560/ad2a97] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/16/2023] [Accepted: 02/19/2024] [Indexed: 02/21/2024]
Abstract
Objective.Accurate delineation of organs-at-risk (OARs) is a critical step in radiotherapy. The deep learning generated segmentations usually need to be reviewed and corrected by oncologists manually, which is time-consuming and operator-dependent. Therefore, an automated quality assurance (QA) and adaptive optimization correction strategy was proposed to identify and optimize 'incorrect' auto-segmentations.Approach.A total of 586 CT images and labels from nine institutions were used. The OARs included the brainstem, parotid, and mandible. The deep learning generated contours were compared with the manual ground truth delineations. In this study, we proposed a novel contour quality assurance and adaptive optimization (CQA-AO) strategy, which consists of the following three main components: (1) the contour QA module classified the deep learning generated contours as either accepted or unaccepted; (2) the unacceptable contour categories analysis module provided the potential error reasons (five unacceptable category) and locations (attention heatmaps); (3) the adaptive correction of unacceptable contours module integrate vision-language representations and utilize convex optimization algorithms to achieve adaptive correction of 'incorrect' contours.Main results. In the contour QA tasks, the sensitivity (accuracy, precision) of CQA-AO strategy reached 0.940 (0.945, 0.948), 0.962 (0.937, 0.913), and 0.967 (0.962, 0.957) for brainstem, parotid and mandible, respectively. The unacceptable contour category analysis, the(FI,AccI,Fmicro,Fmacro)of CQA-AO strategy reached (0.901, 0.763, 0.862, 0.822), (0.855, 0.737, 0.837, 0.784), and (0.907, 0.762, 0.858, 0.821) for brainstem, parotid and mandible, respectively. After adaptive optimization correction, the DSC values of brainstem, parotid and mandible have been improved by 9.4%, 25.9%, and 13.5%, and Hausdorff distance values decreased by 62%, 70.6%, and 81.6%, respectively.Significance. The proposed CQA-AO strategy, which combines QA of contour and adaptive optimization correction for OARs contouring, demonstrated superior performance compare to conventional methods. This method can be implemented in the clinical contouring procedures and improve the efficiency of delineating and reviewing workflow.
Collapse
Affiliation(s)
- Shunyao Luan
- School of Integrated Circuits, Laboratory for Optoelectronics, Huazhong University of Science and Technology, Wuhan, People's Republic of China
| | - Jun Ou-Yang
- School of Integrated Circuits, Laboratory for Optoelectronics, Huazhong University of Science and Technology, Wuhan, People's Republic of China
| | - Xiaofei Yang
- School of Integrated Circuits, Laboratory for Optoelectronics, Huazhong University of Science and Technology, Wuhan, People's Republic of China
| | - Wei Wei
- Department of Radiation Oncology, Hubei Cancer Hospital, TongJi Medical College, Huazhong University of Science and Technology, Wuhan, Hubei, People's Republic of China
| | - Xudong Xue
- Department of Radiation Oncology, Hubei Cancer Hospital, TongJi Medical College, Huazhong University of Science and Technology, Wuhan, Hubei, People's Republic of China
| | - Benpeng Zhu
- School of Integrated Circuits, Laboratory for Optoelectronics, Huazhong University of Science and Technology, Wuhan, People's Republic of China
| |
Collapse
|
3
|
Hanna EM, Sargent E, Hua CH, Merchant TE, Ates O. Performance analysis and knowledge-based quality assurance of critical organ auto-segmentation for pediatric craniospinal irradiation. Sci Rep 2024; 14:4251. [PMID: 38378834 DOI: 10.1038/s41598-024-55015-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/21/2023] [Accepted: 02/19/2024] [Indexed: 02/22/2024] Open
Abstract
Craniospinal irradiation (CSI) is a vital therapeutic approach utilized for young patients suffering from central nervous system disorders such as medulloblastoma. The task of accurately outlining the treatment area is particularly time-consuming due to the presence of several sensitive organs at risk (OAR) that can be affected by radiation. This study aimed to assess two different methods for automating the segmentation process: an atlas technique and a deep learning neural network approach. Additionally, a novel method was devised to prospectively evaluate the accuracy of automated segmentation as a knowledge-based quality assurance (QA) tool. Involving a patient cohort of 100, ranging in ages from 2 to 25 years with a median age of 8, this study employed quantitative metrics centered around overlap and distance calculations to determine the most effective approach for practical clinical application. The contours generated by two distinct methods of atlas and neural network were compared to ground truth contours approved by a radiation oncologist, utilizing 13 distinct metrics. Furthermore, an innovative QA tool was conceptualized, designed for forthcoming cases based on the baseline dataset of 100 patient cases. The calculated metrics indicated that, in the majority of cases (60.58%), the neural network method demonstrated a notably higher alignment with the ground truth. Instances where no difference was observed accounted for 31.25%, while utilization of the atlas method represented 8.17%. The QA tool results showed that the two approaches achieved 100% agreement in 39.4% of instances for the atlas method and in 50.6% of instances for the neural network auto-segmentation. The results indicate that the neural network approach showcases superior performance, and its significantly closer physical alignment to ground truth contours in the majority of cases. The metrics derived from overlap and distance measurements have enabled clinicians to discern the optimal choice for practical clinical application.
Collapse
Affiliation(s)
- Emeline M Hanna
- St. Jude Children's Research Hospital, Memphis, TN, 38105, USA
| | - Emma Sargent
- St. Jude Children's Research Hospital, Memphis, TN, 38105, USA
| | - Chia-Ho Hua
- St. Jude Children's Research Hospital, Memphis, TN, 38105, USA
| | | | - Ozgur Ates
- St. Jude Children's Research Hospital, Memphis, TN, 38105, USA.
| |
Collapse
|
4
|
Maroongroge S, Mohamed ASR, Nguyen C, Guma De la Vega J, Frank SJ, Garden AS, Gunn BG, Lee A, Mayo L, Moreno A, Morrison WH, Phan J, Spiotto MT, Court LE, Fuller CD, Rosenthal DI, Netherton TJ. Clinical acceptability of automatically generated lymph node levels and structures of deglutition and mastication for head and neck radiation therapy. Phys Imaging Radiat Oncol 2024; 29:100540. [PMID: 38356692 PMCID: PMC10864833 DOI: 10.1016/j.phro.2024.100540] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/30/2023] [Revised: 01/22/2024] [Accepted: 01/24/2024] [Indexed: 02/16/2024] Open
Abstract
Background and Purpose Auto-contouring of complex anatomy in computed tomography (CT) scans is a highly anticipated solution to many problems in radiotherapy. In this study, artificial intelligence (AI)-based auto-contouring models were clinically validated for lymph node levels and structures of swallowing and chewing in the head and neck. Materials and Methods CT scans of 145 head and neck radiotherapy patients were retrospectively curated. One cohort (n = 47) was used to analyze seven lymph node levels and the other (n = 98) used to analyze 17 swallowing and chewing structures. Separate nnUnet models were trained and validated using the separate cohorts. For the lymph node levels, preference and clinical acceptability of AI vs human contours were scored. For the swallowing and chewing structures, clinical acceptability was scored. Quantitative analyses of the test sets were performed for AI vs human contours for all structures using overlap and distance metrics. Results Median Dice Similarity Coefficient ranged from 0.77 to 0.89 for lymph node levels and 0.86 to 0.96 for chewing and swallowing structures. The AI contours were superior to or equally preferred to the manual contours at rates ranging from 75% to 91%; there was not a significant difference in clinical acceptability for nodal levels I-V for manual versus AI contours. Across all AI-generated lymph node level contours, 92% were rated as usable with stylistic to no edits. Of the 340 contours in the chewing and swallowing cohort, 4% required minor edits. Conclusions An accurate approach was developed to auto-contour lymph node levels and chewing and swallowing structures on CT images for patients with intact nodal anatomy. Only a small portion of test set auto-contours required minor edits.
Collapse
Affiliation(s)
- Sean Maroongroge
- Department of Radiation Oncology, Division of Radiation Oncology, University of Texas MD Anderson Cancer Center, United States
| | - Abdallah SR. Mohamed
- Department of Radiation Oncology, Division of Radiation Oncology, University of Texas MD Anderson Cancer Center, United States
| | - Callistus Nguyen
- Department of Radiation Physics, Division of Radiation Oncology, University of Texas MD Anderson Cancer Center, United States
| | - Jean Guma De la Vega
- Department of Radiation Physics, Division of Radiation Oncology, University of Texas MD Anderson Cancer Center, United States
| | - Steven J. Frank
- Department of Radiation Oncology, Division of Radiation Oncology, University of Texas MD Anderson Cancer Center, United States
| | - Adam S. Garden
- Department of Radiation Oncology, Division of Radiation Oncology, University of Texas MD Anderson Cancer Center, United States
| | - Brandon G. Gunn
- Department of Radiation Oncology, Division of Radiation Oncology, University of Texas MD Anderson Cancer Center, United States
| | - Anna Lee
- Department of Radiation Oncology, Division of Radiation Oncology, University of Texas MD Anderson Cancer Center, United States
| | - Lauren Mayo
- Department of Radiation Oncology, Division of Radiation Oncology, University of Texas MD Anderson Cancer Center, United States
| | - Amy Moreno
- Department of Radiation Oncology, Division of Radiation Oncology, University of Texas MD Anderson Cancer Center, United States
| | - William H. Morrison
- Department of Radiation Oncology, Division of Radiation Oncology, University of Texas MD Anderson Cancer Center, United States
| | - Jack Phan
- Department of Radiation Oncology, Division of Radiation Oncology, University of Texas MD Anderson Cancer Center, United States
| | - Michael T. Spiotto
- Department of Radiation Oncology, Division of Radiation Oncology, University of Texas MD Anderson Cancer Center, United States
| | - Laurence E. Court
- Department of Radiation Physics, Division of Radiation Oncology, University of Texas MD Anderson Cancer Center, United States
| | - Clifton D. Fuller
- Department of Radiation Oncology, Division of Radiation Oncology, University of Texas MD Anderson Cancer Center, United States
| | - David I. Rosenthal
- Department of Radiation Oncology, Division of Radiation Oncology, University of Texas MD Anderson Cancer Center, United States
| | - Tucker J. Netherton
- Department of Radiation Physics, Division of Radiation Oncology, University of Texas MD Anderson Cancer Center, United States
| |
Collapse
|
5
|
Duan J, Bernard ME, Rong Y, Castle JR, Feng X, Johnson JD, Chen Q. Contour subregion error detection methodology using deep learning auto-segmentation. Med Phys 2023; 50:6673-6683. [PMID: 37793103 DOI: 10.1002/mp.16768] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/26/2023] [Revised: 07/26/2023] [Accepted: 09/17/2023] [Indexed: 10/06/2023] Open
Abstract
BACKGROUND Inaccurate manual organ delineation is one of the high-risk failure modes in radiation treatment. Numerous automated contour quality assurance (QA) systems have been developed to assess contour acceptability; however, manual inspection of flagged cases is a time-consuming and challenging process, and can lead to users overlooking the exact error location. PURPOSE Our aim is to develop and validate a contour QA system that can effectively detect and visualize subregional contour errors, both qualitatively and quantitatively. METHODS/MATERIALS A novel contour subregion error detection (CSED) system was developed using subregional surface distance discrepancies between manual and deep learning auto-segmentation (DLAS) contours. A validation study was conducted using a head and neck public dataset containing 339 cases and evaluated according to knowledge-based pass criteria derived from a clinical training dataset of 60 cases. A blind qualitative evaluation was conducted, comparing the results from the CSED system with manual labels. Subsequently, the CSED-flagged cases were re-examined by a radiation oncologist. RESULTS The CSED system could visualize the diverse types of subregional contour errors qualitatively and quantitatively. In the validation dataset, the CSED system resulted in true positive rates (TPR) of 0.814, 0.800, and 0.771; false positive rates (FPR) of 0.310, 0.267, and 0.298; and accuracies of 0.735, 0.759, and 0.730, for brainstem and left and right parotid contours, respectively. The CSED-assisted manual review caught 13 brainstem, 19 left parotid, and 21 right parotid contour errors missed by conventional human review. The TPR/FPR/accuracy of the CSED-assisted manual review improved to 0.836/0.253/0.784, 0.831/0.171/0.830, and 0.808/0.193/0.807 for each structure, respectively. Further, the time savings achieved through CSED-assisted review improved by 75%, with the time for review taking 24.81 ± 12.84, 26.75 ± 10.41, and 28.71 ± 13.72 s for each structure, respectively. CONCLUSIONS The CSED system enables qualitative and quantitative detection, localization, and visualization of manual segmentation subregional errors utilizing DLAS contours as references. The use of this system has been shown to help reduce the risk of high-risk failure modes resulting from inaccurate organ segmentation.
Collapse
Affiliation(s)
- Jingwei Duan
- Department of Radiation Medicine, University of Kentucky, Lexington, Kentucky, USA
| | - Mark E Bernard
- Department of Radiation Medicine, University of Kentucky, Lexington, Kentucky, USA
| | - Yi Rong
- Department of Radiation Oncology, Mayo Clinic, Phoenix, Arizona, USA
| | - James R Castle
- Department of Radiation Medicine, University of Kentucky, Lexington, Kentucky, USA
| | - Xue Feng
- Carina Medical LLC, Lexington, Kentucky, USA
| | - Jeremiah D Johnson
- Department of Radiation Medicine, University of Kentucky, Lexington, Kentucky, USA
| | - Quan Chen
- Department of Radiation Medicine, University of Kentucky, Lexington, Kentucky, USA
- Department of Radiation Oncology, City of Hope Comprehensive Cancer Center, Duarte, California, USA
| |
Collapse
|
6
|
Duan J, Bernard ME, Castle JR, Feng X, Wang C, Kenamond MC, Chen Q. Contouring quality assurance methodology based on multiple geometric features against deep learning auto-segmentation. Med Phys 2023; 50:2715-2732. [PMID: 36788735 PMCID: PMC10175153 DOI: 10.1002/mp.16299] [Citation(s) in RCA: 6] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/13/2022] [Revised: 01/06/2023] [Accepted: 01/26/2023] [Indexed: 02/16/2023] Open
Abstract
BACKGROUND Contouring error is one of the top failure modes in radiation treatment. Multiple efforts have been made to develop tools to automatically detect segmentation errors. Deep learning-based auto-segmentation (DLAS) has been used as a baseline for flagging manual segmentation errors, but those efforts are limited to using only one or two contour comparison metrics. PURPOSE The purpose of this research is to develop an improved contouring quality assurance system to identify and flag manual contouring errors. METHODS AND MATERIALS DLAS contours were used as a reference to compare with manually segmented contours. A total of 27 geometric agreement metrics were determined from the comparisons between the two segmentation approaches. Feature selection was performed to optimize the training of a machine learning classification model to identify potential contouring errors. A public dataset with 339 cases was used to train and test the classifier. Four independent classifiers were trained using five-fold cross validation, and the predictions from each classifier were ensembled using soft voting. The trained model was validated on a held-out testing dataset. An additional independent clinical dataset with 60 cases was used to test the generalizability of the model. Model predictions were reviewed by an expert to confirm or reject the findings. RESULTS The proposed machine learning multiple features (ML-MF) approach outperformed traditional nonmachine-learning-based approaches that are based on only one or two geometric agreement metrics. The machine learning model achieved recall (precision) values of 0.842 (0.899), 0.762 (0.762), 0.727 (0.842), and 0.773 (0.773) for Brainstem, Parotid_L, Parotid_R, and mandible contours, respectively compared to 0.526 (0.909), 0.619 (0.765), 0.682 (0.882), 0.773 (0.568) for an approach based solely on Dice similarity coefficient values. In the external validation dataset, 66.7, 93.3, 94.1, and 58.8% of flagged cases were confirmed to have contouring errors by an expert for Brainstem, Parotid_L, Parotid_R, and mandible contours, respectively. CONCLUSIONS The proposed ML-MF approach, which includes multiple geometric agreement metrics to flag manual contouring errors, demonstrated superior performance in comparison to traditional methods. This method is easy to implement in clinical practice and can help to reduce the significant time and labor costs associated with manual segmentation and review.
Collapse
Affiliation(s)
- Jingwei Duan
- Department of Radiation Medicine, University of Kentucky, Lexington, KY 40506
| | - Mark E. Bernard
- Department of Radiation Medicine, University of Kentucky, Lexington, KY 40506
| | - James R. Castle
- Carina Medical LLC, 145 Graham Ave, A168, Lexington, KY 40506
| | - Xue Feng
- Carina Medical LLC, 145 Graham Ave, A168, Lexington, KY 40506
| | - Chi Wang
- Department of Internal Medicine, University of Kentucky, Lexington, KY 40506
| | - Mark C. Kenamond
- Department of Radiation Medicine, University of Kentucky, Lexington, KY 40506
| | - Quan Chen
- Department of Radiation Medicine, University of Kentucky, Lexington, KY 40506
| |
Collapse
|
7
|
Luan S, Xue X, Wei C, Ding Y, Zhu B, Wei W. Machine Learning-Based Quality Assurance for Automatic Segmentation of Head-and-Neck Organs-at-Risk in Radiotherapy. Technol Cancer Res Treat 2023; 22:15330338231157936. [PMID: 36788411 PMCID: PMC9932790 DOI: 10.1177/15330338231157936] [Citation(s) in RCA: 5] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/16/2023] Open
Abstract
Purpose/Objective(s): With the development of deep learning, more convolutional neural networks (CNNs) are being introduced in automatic segmentation to reduce oncologists' labor requirement. However, it is still challenging for oncologists to spend considerable time evaluating the quality of the contours generated by the CNNs. Besides, all the evaluation criteria, such as Dice Similarity Coefficient (DSC), need a gold standard to assess the quality of the contours. To address these problems, we propose an automatic quality assurance (QA) method using isotropic and anisotropic methods to automatically analyze contour quality without a gold standard. Materials/Methods: We used 196 individuals with 18 different head-and-neck organs-at-risk. The overall process has the following 4 main steps. (1) Use CNN segmentation network to generate a series of contours, then use these contours as organ masks to erode and dilate to generate inner/outer shells for each 2D slice. (2) Thirty-eight radiomics features were extracted from these 2 shells, using the inner/outer shells' radiomics features ratios and DSCs as the input for 12 machine learning models. (3) Using the DSC threshold adaptively classified the passing/un-passing slices. (4) Through 2 different threshold analysis methods quantitatively evaluated the un-passing slices and obtained a series of location information of poor contours. Parts 1-3 were isotropic experiments, and part 4 was the anisotropic method. Result: From the isotropic experiments, almost all the predicted values were close to the labels. Through the anisotropic method, we obtained the contours' location information by assessing the thresholds of the peak-to-peak and area-to-area ratios. Conclusion: The proposed automatic segmentation QA method could predict the segmentation quality qualitatively. Moreover, the method can analyze the location information for un-passing slices.
Collapse
Affiliation(s)
- Shunyao Luan
- Department of Radiation Oncology, Hubei Cancer
Hospital, Tongji Medical College, Huazhong
University of Science and Technology, Wuhan, China,School of Optical and Electronic Information,
Huazhong
University of Science and Technology,
Wuhan, China
| | - Xudong Xue
- Department of Radiation Oncology, Hubei Cancer
Hospital, Tongji Medical College, Huazhong
University of Science and Technology, Wuhan, China
| | - Changchao Wei
- Department of Radiation Oncology, Hubei Cancer
Hospital, Tongji Medical College, Huazhong
University of Science and Technology, Wuhan, China,Key Laboratory of Artificial Micro and Nano-structures of Ministry
of Education, Center for Theoretical Physics, School of Physics and Technology,
Wuhan
University, Wuhan, China
| | - Yi Ding
- Department of Radiation Oncology, Hubei Cancer
Hospital, Tongji Medical College, Huazhong
University of Science and Technology, Wuhan, China
| | - Benpeng Zhu
- School of Optical and Electronic Information,
Huazhong
University of Science and Technology,
Wuhan, China,Benpeng Zhu, School of Optical and
Electronic Information, Huazhong University of Science and Technology, Wuhan,
430000, China.
| | - Wei Wei
- Department of Radiation Oncology, Hubei Cancer
Hospital, Tongji Medical College, Huazhong
University of Science and Technology, Wuhan, China,Wei Wei, Department of Radiation Oncology,
Hubei Cancer Hospital, Tongji Medical College, Huazhong University of Science
and Technology, Wuhan, 430079, China.
| |
Collapse
|
8
|
Pera Ó, Martínez Á, Möhler C, Hamans B, Vega F, Barral F, Becerra N, Jimenez R, Fernandez-Velilla E, Quera J, Algara M. Clinical Validation of Siemens' Syngo.via Automatic Contouring System. Adv Radiat Oncol 2023; 8:101177. [PMID: 36865668 PMCID: PMC9972393 DOI: 10.1016/j.adro.2023.101177] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/28/2022] [Accepted: 01/05/2023] [Indexed: 01/18/2023] Open
Abstract
Purpose The manual delineation of organs at risk is a process that requires a great deal of time both for the technician and for the physician. Availability of validated software tools assisted by artificial intelligence would be of great benefit, as it would significantly improve the radiation therapy workflow, reducing the time required for segmentation. The purpose of this article is to validate the deep learning-based autocontouring solution integrated in syngo.via RT Image Suite VB40 (Siemens Healthineers, Forchheim, Germany). Methods and Materials For this purpose, we have used our own specific qualitative classification system, RANK, to evaluate more than 600 contours corresponding to 18 different automatically delineated organs at risk. Computed tomography data sets of 95 different patients were included: 30 patients with lung, 30 patients with breast, and 35 male patients with pelvic cancer. The automatically generated structures were reviewed in the Eclipse Contouring module independently by 3 observers: an expert physician, an expert technician, and a junior physician. Results There is a statistically significant difference between the Dice coefficient associated with RANK 4 compared with the coefficient associated with RANKs 2 and 3 (P < .001). In total, 64% of the evaluated structures received the maximum score, 4. Only 1% of the structures were classified with the lowest score, 1. The time savings for breast, thorax, and pelvis were 87.6%, 93.5%, and 82.2%, respectively. Conclusions Siemens' syngo.via RT Image Suite offers good autocontouring results and significant time savings.
Collapse
Affiliation(s)
- Óscar Pera
- Radiation Oncology Department, Hospital del Mar, Barcelona, Spain,Institut Hospital del Mar d'Investigacions Mèdiques, Barcelona, Spain,Corresponding author: Óscar Pera, MSc
| | - Álvaro Martínez
- Radiation Oncology Department, Hospital del Mar, Barcelona, Spain
| | | | | | | | | | - Nuria Becerra
- Radiation Oncology Department, Hospital del Mar, Barcelona, Spain
| | - Rafael Jimenez
- Radiation Oncology Department, Hospital del Mar, Barcelona, Spain
| | - Enric Fernandez-Velilla
- Radiation Oncology Department, Hospital del Mar, Barcelona, Spain,Institut Hospital del Mar d'Investigacions Mèdiques, Barcelona, Spain
| | - Jaume Quera
- Radiation Oncology Department, Hospital del Mar, Barcelona, Spain,Institut Hospital del Mar d'Investigacions Mèdiques, Barcelona, Spain
| | - Manuel Algara
- Radiation Oncology Department, Hospital del Mar, Barcelona, Spain,Institut Hospital del Mar d'Investigacions Mèdiques, Barcelona, Spain,Autonomous University of Barcelona, Barcelona, Spain
| |
Collapse
|
9
|
Strolin S, Santoro M, Paolani G, Ammendolia I, Arcelli A, Benini A, Bisello S, Cardano R, Cavallini L, Deraco E, Donati CM, Galietta E, Galuppi A, Guido A, Ferioli M, Laghi V, Medici F, Ntreta M, Razganiayeva N, Siepe G, Tolento G, Vallerossa D, Zamagni A, Morganti AG, Strigari L. How smart is artificial intelligence in organs delineation? Testing a CE and FDA-approved Deep-Learning tool using multiple expert contours delineated on planning CT images. Front Oncol 2023; 13:1089807. [PMID: 36937399 PMCID: PMC10019504 DOI: 10.3389/fonc.2023.1089807] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/04/2022] [Accepted: 01/19/2023] [Indexed: 03/06/2023] Open
Abstract
Background A CE- and FDA-approved cloud-based Deep learning (DL)-tool for automatic organs at risk (OARs) and clinical target volumes segmentation on computer tomography images is available. Before its implementation in the clinical practice, an independent external validation was conducted. Methods At least a senior and two in training Radiation Oncologists (ROs) manually contoured the volumes of interest (VOIs) for 6 tumoral sites. The auto-segmented contours were retrieved from the DL-tool and, if needed, manually corrected by ROs. The level of ROs satisfaction and the duration of contouring were registered. Relative volume differences, similarity indices, satisfactory grades, and time saved were analyzed using a semi-automatic tool. Results Seven thousand seven hundred sixty-five VOIs were delineated on the CT images of 111 representative patients. The median (range) time for manual VOIs delineation, DL-based segmentation, and subsequent manual corrections were 25.0 (8.0-115.0), 2.3 (1.2-8) and 10.0 minutes (0.3-46.3), respectively. The overall time for VOIs retrieving and modification was statistically significantly lower than for manual contouring (p<0.001). The DL-tool was generally appreciated by ROs, with 44% of vote 4 (well done) and 43% of vote 5 (very well done), correlated with the saved time (p<0.001). The relative volume differences and similarity indexes suggested a better inter-agreement of manually adjusted DL-based VOIs than manually segmented ones. Conclusions The application of the DL-tool resulted satisfactory, especially in complex delineation cases, improving the ROs inter-agreement of delineated VOIs and saving time.
Collapse
Affiliation(s)
- Silvia Strolin
- Department of Medical Physics, IRCCS Azienda Ospedaliero-Universitaria di Bologna, Bologna, Italy
| | - Miriam Santoro
- Department of Medical Physics, IRCCS Azienda Ospedaliero-Universitaria di Bologna, Bologna, Italy
- Medical Physics Specialization School, Alma Mater Studiorum, University of Bologna, Bologna, Italy
| | - Giulia Paolani
- Department of Medical Physics, IRCCS Azienda Ospedaliero-Universitaria di Bologna, Bologna, Italy
- Medical Physics Specialization School, Alma Mater Studiorum, University of Bologna, Bologna, Italy
- *Correspondence: Lidia Strigari, ; Giulia Paolani,
| | - Ilario Ammendolia
- Department of Radiation Oncology, IRCCS Azienda Ospedaliero-Universitaria di Bologna, Bologna, Italy
| | - Alessandra Arcelli
- Department of Radiation Oncology, IRCCS Azienda Ospedaliero-Universitaria di Bologna, Bologna, Italy
| | - Anna Benini
- Department of Radiation Oncology, IRCCS Azienda Ospedaliero-Universitaria di Bologna, Bologna, Italy
| | - Silvia Bisello
- Department of Radiation Oncology, IRCCS Azienda Ospedaliero-Universitaria di Bologna, Bologna, Italy
| | - Raffaele Cardano
- Department of Radiation Oncology, IRCCS Azienda Ospedaliero-Universitaria di Bologna, Bologna, Italy
| | - Letizia Cavallini
- Department of Radiation Oncology, IRCCS Azienda Ospedaliero-Universitaria di Bologna, Bologna, Italy
| | - Elisa Deraco
- Department of Radiation Oncology, IRCCS Azienda Ospedaliero-Universitaria di Bologna, Bologna, Italy
| | - Costanza Maria Donati
- Department of Radiation Oncology, IRCCS Azienda Ospedaliero-Universitaria di Bologna, Bologna, Italy
| | - Erika Galietta
- Department of Radiation Oncology, IRCCS Azienda Ospedaliero-Universitaria di Bologna, Bologna, Italy
| | - Andrea Galuppi
- Department of Radiation Oncology, IRCCS Azienda Ospedaliero-Universitaria di Bologna, Bologna, Italy
| | - Alessandra Guido
- Department of Radiation Oncology, IRCCS Azienda Ospedaliero-Universitaria di Bologna, Bologna, Italy
| | - Martina Ferioli
- Department of Radiation Oncology, IRCCS Azienda Ospedaliero-Universitaria di Bologna, Bologna, Italy
| | - Viola Laghi
- Department of Radiation Oncology, IRCCS Azienda Ospedaliero-Universitaria di Bologna, Bologna, Italy
| | - Federica Medici
- Department of Radiation Oncology, IRCCS Azienda Ospedaliero-Universitaria di Bologna, Bologna, Italy
| | - Maria Ntreta
- Department of Radiation Oncology, IRCCS Azienda Ospedaliero-Universitaria di Bologna, Bologna, Italy
| | - Natalya Razganiayeva
- Department of Radiation Oncology, IRCCS Azienda Ospedaliero-Universitaria di Bologna, Bologna, Italy
| | - Giambattista Siepe
- Department of Radiation Oncology, IRCCS Azienda Ospedaliero-Universitaria di Bologna, Bologna, Italy
| | - Giorgio Tolento
- Department of Radiation Oncology, IRCCS Azienda Ospedaliero-Universitaria di Bologna, Bologna, Italy
| | - Daria Vallerossa
- Department of Radiation Oncology, IRCCS Azienda Ospedaliero-Universitaria di Bologna, Bologna, Italy
| | - Alice Zamagni
- Department of Radiation Oncology, IRCCS Azienda Ospedaliero-Universitaria di Bologna, Bologna, Italy
| | - Alessio Giuseppe Morganti
- Department of Radiation Oncology, IRCCS Azienda Ospedaliero-Universitaria di Bologna, Bologna, Italy
- Department of Medical and Surgical Sciences (DIMEC), Alma Mater Studiorum, University of Bologna, Bologna, Italy
| | - Lidia Strigari
- Department of Medical Physics, IRCCS Azienda Ospedaliero-Universitaria di Bologna, Bologna, Italy
- *Correspondence: Lidia Strigari, ; Giulia Paolani,
| |
Collapse
|
10
|
Isaksson LJ, Summers P, Bhalerao A, Gandini S, Raimondi S, Pepa M, Zaffaroni M, Corrao G, Mazzola GC, Rotondi M, Lo Presti G, Haron Z, Alessi S, Pricolo P, Mistretta FA, Luzzago S, Cattani F, Musi G, De Cobelli O, Cremonesi M, Orecchia R, Marvaso G, Petralia G, Jereczek-Fossa BA. Quality assurance for automatically generated contours with additional deep learning. Insights Imaging 2022; 13:137. [PMID: 35976491 PMCID: PMC9385913 DOI: 10.1186/s13244-022-01276-7] [Citation(s) in RCA: 6] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/30/2022] [Accepted: 07/24/2022] [Indexed: 11/10/2022] Open
Abstract
OBJECTIVE Deploying an automatic segmentation model in practice should require rigorous quality assurance (QA) and continuous monitoring of the model's use and performance, particularly in high-stakes scenarios such as healthcare. Currently, however, tools to assist with QA for such models are not available to AI researchers. In this work, we build a deep learning model that estimates the quality of automatically generated contours. METHODS The model was trained to predict the segmentation quality by outputting an estimate of the Dice similarity coefficient given an image contour pair as input. Our dataset contained 60 axial T2-weighted MRI images of prostates with ground truth segmentations along with 80 automatically generated segmentation masks. The model we used was a 3D version of the EfficientDet architecture with a custom regression head. For validation, we used a fivefold cross-validation. To counteract the limitation of the small dataset, we used an extensive data augmentation scheme capable of producing virtually infinite training samples from a single ground truth label mask. In addition, we compared the results against a baseline model that only uses clinical variables for its predictions. RESULTS Our model achieved a mean absolute error of 0.020 ± 0.026 (2.2% mean percentage error) in estimating the Dice score, with a rank correlation of 0.42. Furthermore, the model managed to correctly identify incorrect segmentations (defined in terms of acceptable/unacceptable) 99.6% of the time. CONCLUSION We believe that the trained model can be used alongside automatic segmentation tools to ensure quality and thus allow intervention to prevent undesired segmentation behavior.
Collapse
Affiliation(s)
| | - Paul Summers
- Division of Radiology, IEO European Institute of Oncology IRCCS, Milan, Italy
| | - Abhir Bhalerao
- Department of Computer Science, University of Warwick, Coventry, Warwick, CV4 7AL, UK
| | - Sara Gandini
- Molecular and Pharmaco-Epidemiology Unit, Department of Experimental Oncology, IEO European Institute of Oncology IRCCS, Milan, Italy
| | - Sara Raimondi
- Molecular and Pharmaco-Epidemiology Unit, Department of Experimental Oncology, IEO European Institute of Oncology IRCCS, Milan, Italy
| | - Matteo Pepa
- Division of Radiation Oncology, IEO European Institute of Oncology IRCCS, Milan, Italy
| | - Mattia Zaffaroni
- Division of Radiation Oncology, IEO European Institute of Oncology IRCCS, Milan, Italy
| | - Giulia Corrao
- Division of Radiation Oncology, IEO European Institute of Oncology IRCCS, Milan, Italy.,Department of Oncology and Hemato-Oncology, University of Milan, Milan, Italy
| | - Giovanni Carlo Mazzola
- Division of Radiation Oncology, IEO European Institute of Oncology IRCCS, Milan, Italy.,Department of Oncology and Hemato-Oncology, University of Milan, Milan, Italy
| | - Marco Rotondi
- Division of Radiation Oncology, IEO European Institute of Oncology IRCCS, Milan, Italy.,Department of Oncology and Hemato-Oncology, University of Milan, Milan, Italy
| | - Giuliana Lo Presti
- Molecular and Pharmaco-Epidemiology Unit, Department of Experimental Oncology, IEO European Institute of Oncology IRCCS, Milan, Italy
| | - Zaharudin Haron
- Radiology Department, National Cancer Institute, Putrajaya, Malaysia
| | - Sara Alessi
- Division of Radiology, IEO European Institute of Oncology IRCCS, Milan, Italy
| | - Paola Pricolo
- Division of Radiology, IEO European Institute of Oncology IRCCS, Milan, Italy
| | | | - Stefano Luzzago
- Division of Urology, IEO European Institute of Oncology IRCCS, Milan, Italy
| | - Federica Cattani
- Medical Physics Unit, IEO European Institute of Oncology IRCCS, Milan, Italy
| | - Gennaro Musi
- Department of Oncology and Hemato-Oncology, University of Milan, Milan, Italy.,Division of Urology, IEO European Institute of Oncology IRCCS, Milan, Italy
| | - Ottavio De Cobelli
- Department of Oncology and Hemato-Oncology, University of Milan, Milan, Italy.,Division of Urology, IEO European Institute of Oncology IRCCS, Milan, Italy
| | - Marta Cremonesi
- Radiation Research Unit, IEO European Institute of Oncology IRCCS, Milan, Italy
| | - Roberto Orecchia
- Scientific Direction, IEO European Institute of Oncology IRCCS, Milan, Italy
| | - Giulia Marvaso
- Division of Radiation Oncology, IEO European Institute of Oncology IRCCS, Milan, Italy.,Department of Oncology and Hemato-Oncology, University of Milan, Milan, Italy
| | - Giuseppe Petralia
- Department of Oncology and Hemato-Oncology, University of Milan, Milan, Italy.,Precision Imaging and Research Unit, Department of Medical Imaging and Radiation Sciences, IEO European Institute of Oncology IRCCS, Milan, Italy
| | - Barbara Alicja Jereczek-Fossa
- Division of Radiation Oncology, IEO European Institute of Oncology IRCCS, Milan, Italy.,Department of Oncology and Hemato-Oncology, University of Milan, Milan, Italy
| |
Collapse
|
11
|
Im JH, Lee IJ, Choi Y, Sung J, Ha JS, Lee H. Impact of Denoising on Deep-Learning-Based Automatic Segmentation Framework for Breast Cancer Radiotherapy Planning. Cancers (Basel) 2022; 14:cancers14153581. [PMID: 35892839 PMCID: PMC9332287 DOI: 10.3390/cancers14153581] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/02/2022] [Revised: 07/08/2022] [Accepted: 07/20/2022] [Indexed: 02/04/2023] Open
Abstract
Objective: This study aimed to investigate the segmentation accuracy of organs at risk (OARs) when denoised computed tomography (CT) images are used as input data for a deep-learning-based auto-segmentation framework. Methods: We used non-contrast enhanced planning CT scans from 40 patients with breast cancer. The heart, lungs, esophagus, spinal cord, and liver were manually delineated by two experienced radiation oncologists in a double-blind manner. The denoised CT images were used as input data for the AccuContourTM segmentation software to increase the signal difference between structures of interest and unwanted noise in non-contrast CT. The accuracy of the segmentation was assessed using the Dice similarity coefficient (DSC), and the results were compared with those of conventional deep-learning-based auto-segmentation without denoising. Results: The average DSC outcomes were higher than 0.80 for all OARs except for the esophagus. AccuContourTM-based and denoising-based auto-segmentation demonstrated comparable performance for the lungs and spinal cord but showed limited performance for the esophagus. Denoising-based auto-segmentation for the liver was minimal but had statistically significantly better DSC than AccuContourTM-based auto-segmentation (p < 0.05). Conclusions: Denoising-based auto-segmentation demonstrated satisfactory performance in automatic liver segmentation from non-contrast enhanced CT scans. Further external validation studies with larger cohorts are needed to verify the usefulness of denoising-based auto-segmentation.
Collapse
Affiliation(s)
- Jung Ho Im
- CHA Bundang Medical Center, Department of Radiation Oncology, CHA University School of Medicine, Seongnam 13496, Korea;
| | - Ik Jae Lee
- Department of Radiation Oncology, Yonsei University College of Medicine, Seoul 03722, Korea; (I.J.L.); (J.S.)
| | - Yeonho Choi
- Department of Radiation Oncology, Gangnam Severance Hospital, Seoul 06273, Korea; (Y.C.); (J.S.H.)
| | - Jiwon Sung
- Department of Radiation Oncology, Yonsei University College of Medicine, Seoul 03722, Korea; (I.J.L.); (J.S.)
| | - Jin Sook Ha
- Department of Radiation Oncology, Gangnam Severance Hospital, Seoul 06273, Korea; (Y.C.); (J.S.H.)
| | - Ho Lee
- Department of Radiation Oncology, Yonsei University College of Medicine, Seoul 03722, Korea; (I.J.L.); (J.S.)
- Correspondence: ; Tel.: +82-2-2228-8109; Fax: +82-2-2227-7823
| |
Collapse
|
12
|
Recent Applications of Artificial Intelligence in Radiotherapy: Where We Are and Beyond. APPLIED SCIENCES-BASEL 2022. [DOI: 10.3390/app12073223] [Citation(s) in RCA: 10] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/12/2022]
Abstract
In recent decades, artificial intelligence (AI) tools have been applied in many medical fields, opening the possibility of finding novel solutions for managing very complex and multifactorial problems, such as those commonly encountered in radiotherapy (RT). We conducted a PubMed and Scopus search to identify the AI application field in RT limited to the last four years. In total, 1824 original papers were identified, and 921 were analyzed by considering the phase of the RT workflow according to the applied AI approaches. AI permits the processing of large quantities of information, data, and images stored in RT oncology information systems, a process that is not manageable for individuals or groups. AI allows the iterative application of complex tasks in large datasets (e.g., delineating normal tissues or finding optimal planning solutions) and might support the entire community working in the various sectors of RT, as summarized in this overview. AI-based tools are now on the roadmap for RT and have been applied to the entire workflow, mainly for segmentation, the generation of synthetic images, and outcome prediction. Several concerns were raised, including the need for harmonization while overcoming ethical, legal, and skill barriers.
Collapse
|
13
|
Lee SH, Geng H, Xiao Y. Radiotherapy Standardisation and Artificial Intelligence within the National Cancer Institute's Clinical Trials Network. Clin Oncol (R Coll Radiol) 2022; 34:128-134. [PMID: 34906407 PMCID: PMC8792288 DOI: 10.1016/j.clon.2021.11.020] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/17/2021] [Revised: 11/09/2021] [Accepted: 11/25/2021] [Indexed: 02/03/2023]
Abstract
Artificial intelligence in healthcare refers to the use of complex algorithms designed to conduct certain tasks in an automated manner. Artificial intelligence has a transformative power in radiation oncology to improve the quality and efficiency of patient care, given the increase in volume and complexity of digital data, as well as the multi-faceted and highly technical nature of this field of medicine. However, artificial intelligence alone will not be able to fix healthcare's problem, because new technologies bring unexpected and potentially underappreciated obstacles. The inclusion of multicentre datasets, the incorporation of time-varying data, the assessment of missing data as well as informative censoring and the addition of clinical utility could significantly improve artificial intelligence models. Standardisation plays a crucial, supportive and leading role in artificial intelligence. Clinical trials are the most reliable method of demonstrating the efficacy and safety of a treatment or clinical approach, as well as providing high-level evidence to justify artificial intelligence. The National Surgical Adjuvant Breast and Bowel Project, the Radiation Therapy Oncology Group and the Gynecologic Oncology Group collaborated to form NRG Oncology (acronym NRG derived from the names of the parental groups). NRG Oncology is one of the adult cancer clinical trial groups containing radiotherapy specialty of the National Cancer Institute's Clinical Trials Network (NCTN). Standardisation from NRG/NCTN has the potential to reduce variation in clinical treatment and patient outcome by eliminating potential errors, enabling broader application of artificial intelligence tools. NCTN, NRG and Imaging and Radiation Oncology Core are in a unique position to help with standards development, advocacy and enforcement, all of which can benefit from artificial intelligence, as artificial intelligence has the ability to improve trial success rates by transforming crucial phases in clinical trial design, from study planning through to execution. Here we will examine: (i) how to conduct technical and clinical evaluations before adopting artificial intelligence technologies, (ii) how to obtain high-quality data for artificial intelligence, (iii) the NCTN infrastructure and standards, (iv) radiotherapy standardisation for clinical trials and (v) artificial intelligence applications in standardisation.
Collapse
Affiliation(s)
- S H Lee
- Department of Radiation Oncology, University of Pennsylvania, Philadelphia, Pennsylvania, USA.
| | - H Geng
- Department of Radiation Oncology, University of Pennsylvania, Philadelphia, Pennsylvania, USA.
| | - Y Xiao
- Department of Radiation Oncology, University of Pennsylvania, Philadelphia, Pennsylvania, USA.
| |
Collapse
|
14
|
Min H, Dowling J, Jameson MG, Cloak K, Faustino J, Sidhom M, Martin J, Ebert MA, Haworth A, Chlap P, de Leon J, Berry M, Pryor D, Greer P, Vinod SK, Holloway L. Automatic radiotherapy delineation quality assurance on prostate MRI with deep learning in a multicentre clinical trial. Phys Med Biol 2021; 66. [PMID: 34507305 DOI: 10.1088/1361-6560/ac25d5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/16/2021] [Accepted: 09/10/2021] [Indexed: 11/11/2022]
Abstract
Volume delineation quality assurance (QA) is particularly important in clinical trial settings where consistent protocol implementation is required, as outcomes will affect future as well current patients. Currently, where feasible, this is conducted manually, which is time consuming and resource intensive. Although previous studies mostly focused on automating delineation QA on CT, magnetic resonance imaging (MRI) is being increasingly used in radiotherapy treatment. In this work, we propose to perform automatic delineation QA on prostate MRI for both the clinical target volume (CTV) and organs-at-risk (OARs) by using delineations generated by 3D Unet variants as benchmarks for QA. These networks were trained on a small gold standard atlas set and applied on a multicentre radiotherapy clinical trial dataset to generate benchmark delineations. Then, a QA stage was designed to recommend 'pass', 'minor correction' and 'major correction' for each manual delineation in the trial set by thresholding its Dice similarity coefficient to the network generated delineation. Among all 3D Unet variants explored, the Unet with anatomical gates in an AtlasNet architecture performed the best in delineation QA, achieving an area under the receiver operating characteristics curve of 0.97, 0.92, 0.89 and 0.97 for identifying unacceptable (major correction) delineations with a sensitivity of 0.93, 0.73, 0.74 and 0.90 at a specificity of 0.93, 0.86, 0.86 and 0.95 for bladder, prostate CTV, rectum and gel spacer respectively. To the best of our knowledge, this is the first study to propose automated delineation QA for a multicentre radiotherapy clinical trial with treatment planning MRI. The methods proposed in this work can potentially improve the accuracy and consistency of CTV and OAR delineation in radiotherapy treatment planning.
Collapse
Affiliation(s)
- Hang Min
- CSIRO Australian e-Health Research Centre, Herston, Queensland, Australia.,Ingham Institute for Applied Medical Research, Sydney, New South Wales, Australia.,South Western Clinical School, University of New South Wales, Australia
| | - Jason Dowling
- CSIRO Australian e-Health Research Centre, Herston, Queensland, Australia.,South Western Clinical School, University of New South Wales, Australia.,Centre for Medical Radiation Physics, University of Wollongong, New South Wales, Australia.,Institute of Medical Physics, The University of Sydney, New South Wales, Australia.,School of Mathematical and Physical Sciences, University of Newcastle, New South Wales, Australia
| | - Michael G Jameson
- St Vincent's Clinical School, Faculty of Medicine, University of New South Wales, Australia.,GenesisCare, Sydney, New South Wales, Australia
| | - Kirrily Cloak
- Ingham Institute for Applied Medical Research, Sydney, New South Wales, Australia.,South Western Clinical School, University of New South Wales, Australia
| | - Joselle Faustino
- Liverpool and Macarthur Cancer therapy Centres, Liverpool Hospital, New South Wales, Australia
| | - Mark Sidhom
- South Western Clinical School, University of New South Wales, Australia.,Liverpool and Macarthur Cancer therapy Centres, Liverpool Hospital, New South Wales, Australia
| | - Jarad Martin
- Department of Radiation Oncology, Calvary Mater Newcastle, Newcastle, New South Wales, Australia
| | - Martin A Ebert
- Centre for Medical Radiation Physics, University of Wollongong, New South Wales, Australia.,Department of Radiation Oncology, Sir Charles Gairdner Hospital, Nedlands, Western Australia, Australia.,School of Physics Mathematics and Computing, University of Western Australia, Perth, Western Australia, Australia
| | - Annette Haworth
- Institute of Medical Physics, The University of Sydney, New South Wales, Australia
| | - Phillip Chlap
- Ingham Institute for Applied Medical Research, Sydney, New South Wales, Australia.,South Western Clinical School, University of New South Wales, Australia.,Liverpool and Macarthur Cancer therapy Centres, Liverpool Hospital, New South Wales, Australia
| | - Jeremiah de Leon
- GenesisCare, Sydney, New South Wales, Australia.,Illawarra Cancer Care Centre, Wollongong, Australia
| | - Megan Berry
- South Western Clinical School, University of New South Wales, Australia.,Liverpool and Macarthur Cancer therapy Centres, Liverpool Hospital, New South Wales, Australia
| | - David Pryor
- Princess Alexandra Hospital, Brisbane, Queensland, Australia
| | - Peter Greer
- School of Mathematical and Physical Sciences, University of Newcastle, New South Wales, Australia.,Department of Radiation Oncology, Calvary Mater Newcastle, Newcastle, New South Wales, Australia
| | - Shalini K Vinod
- Ingham Institute for Applied Medical Research, Sydney, New South Wales, Australia.,South Western Clinical School, University of New South Wales, Australia.,Liverpool and Macarthur Cancer therapy Centres, Liverpool Hospital, New South Wales, Australia
| | - Lois Holloway
- Ingham Institute for Applied Medical Research, Sydney, New South Wales, Australia.,South Western Clinical School, University of New South Wales, Australia.,Centre for Medical Radiation Physics, University of Wollongong, New South Wales, Australia.,Institute of Medical Physics, The University of Sydney, New South Wales, Australia.,Liverpool and Macarthur Cancer therapy Centres, Liverpool Hospital, New South Wales, Australia
| |
Collapse
|
15
|
Oya M, Sugimoto S, Sasai K, Yokoyama K. Investigation of clinical target volume segmentation for whole breast irradiation using three-dimensional convolutional neural networks with gradient-weighted class activation mapping. Radiol Phys Technol 2021; 14:238-247. [PMID: 34132994 DOI: 10.1007/s12194-021-00620-8] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/16/2020] [Revised: 05/11/2021] [Accepted: 05/25/2021] [Indexed: 12/22/2022]
Abstract
This study aims to implement three-dimensional convolutional neural networks (3D-CNN) for clinical target volume (CTV) segmentation for whole breast irradiation and investigate the focus of 3D-CNNs during decision-making using gradient-weighted class activation mapping (Grad-CAM). A 3D-UNet CNN was adopted to conduct automatic segmentation of the CTV for breast cancer. The 3D-UNet was trained using three datasets of left-, right-, and both left- and right-sided breast cancer patients. Segmentation accuracy was evaluated using the Dice similarity coefficient (DSC). Grad-CAM was applied to trained CNNs. The DSCs for the datasets of the left-, right-, and both left- and right-sided breasts were on an average 0.88, 0.89, and 0.85, respectively. The Grad-CAM heatmaps showed that the 3D-UNet used for segmentation determined the CTV region from the target-side breast tissue and by referring to the opposite-side breast. Although the size of the dataset was limited, DSC ≥ 0.85 was achieved for the segmentation of breast CTV using the 3D-UNet. Grad-CAM indicates the applicable scope and limitations of using a CNN by indicating the focus of such networks during decision-making.
Collapse
Affiliation(s)
- Megumi Oya
- Department of Epidemiology and Environmental Health, Juntendo University Graduate School of Medicine, 2-1-1 Hongo, Bunkyo-ku, Tokyo, 113-8421, Japan
| | - Satoru Sugimoto
- Department of Radiation Oncology, Juntendo University Graduate School of Medicine, 2-1-1 Hongo, Bunkyo-ku, Tokyo, 113-8421, Japan.
| | - Keisuke Sasai
- Department of Radiation Oncology, Juntendo University Graduate School of Medicine, 2-1-1 Hongo, Bunkyo-ku, Tokyo, 113-8421, Japan
| | - Kazuhito Yokoyama
- Department of Epidemiology and Environmental Health, Juntendo University Graduate School of Medicine, 2-1-1 Hongo, Bunkyo-ku, Tokyo, 113-8421, Japan.,Department of Epidemiology and Social Medicine, International University of Health and Welfare Graduate School of Public Health, 4-1-26 Akasaka, Minato-ku, Tokyo, 107-8402, Japan
| |
Collapse
|
16
|
Brouwer CL, Boukerroui D, Oliveira J, Looney P, Steenbakkers RJ, Langendijk JA, Both S, Gooding MJ. Assessment of manual adjustment performed in clinical practice following deep learning contouring for head and neck organs at risk in radiotherapy. Phys Imaging Radiat Oncol 2020; 16:54-60. [PMID: 33458344 PMCID: PMC7807591 DOI: 10.1016/j.phro.2020.10.001] [Citation(s) in RCA: 22] [Impact Index Per Article: 5.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/05/2020] [Revised: 09/30/2020] [Accepted: 10/01/2020] [Indexed: 12/30/2022] Open
Abstract
BACKGROUND AND PURPOSE Auto-contouring performance has been widely studied in development and commissioning studies in radiotherapy, and its impact on clinical workflow assessed in that context. This study aimed to evaluate the manual adjustment of auto-contouring in routine clinical practice and to identify improvements regarding the auto-contouring model and clinical user interaction, to improve the efficiency of auto-contouring. MATERIALS AND METHODS A total of 103 clinical head and neck cancer cases, contoured using a commercial deep-learning contouring system and subsequently checked and edited for clinical use were retrospectively taken from clinical data over a twelve-month period (April 2019-April 2020). The amount of adjustment performed was calculated, and all cases were registered to a common reference frame for assessment purposes. The median, 10th and 90th percentile of adjustment were calculated and displayed using 3D renderings of structures to visually assess systematic and random adjustment. Results were also compared to inter-observer variation reported previously. Assessment was performed for both the whole structures and for regional sub-structures, and according to the radiation therapy technologist (RTT) who edited the contour. RESULTS The median amount of adjustment was low for all structures (<2 mm), although large local adjustment was observed for some structures. The median was systematically greater or equal to zero, indicating that the auto-contouring tends to under-segment the desired contour. CONCLUSION Auto-contouring performance assessment in routine clinical practice has identified systematic improvements required technically, but also highlighted the need for continued RTT training to ensure adherence to guidelines.
Collapse
Affiliation(s)
- Charlotte L. Brouwer
- University of Groningen, University Medical Center Groningen, Department of Radiation Oncology, Groningen, The Netherlands
| | | | | | | | - Roel J.H.M. Steenbakkers
- University of Groningen, University Medical Center Groningen, Department of Radiation Oncology, Groningen, The Netherlands
| | - Johannes A. Langendijk
- University of Groningen, University Medical Center Groningen, Department of Radiation Oncology, Groningen, The Netherlands
| | - Stefan Both
- University of Groningen, University Medical Center Groningen, Department of Radiation Oncology, Groningen, The Netherlands
| | | |
Collapse
|