1
|
Ren J, Teuwen J, Nijkamp J, Rasmussen M, Gouw Z, Grau Eriksen J, Sonke JJ, Korreman S. Enhancing the reliability of deep learning-based head and neck tumour segmentation using uncertainty estimation with multi-modal images. Phys Med Biol 2024; 69:165018. [PMID: 39059432 DOI: 10.1088/1361-6560/ad682d] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/15/2024] [Accepted: 07/26/2024] [Indexed: 07/28/2024]
Abstract
Objective.Deep learning shows promise in autosegmentation of head and neck cancer (HNC) primary tumours (GTV-T) and nodal metastases (GTV-N). However, errors such as including non-tumour regions or missing nodal metastases still occur. Conventional methods often make overconfident predictions, compromising reliability. Incorporating uncertainty estimation, which provides calibrated confidence intervals can address this issue. Our aim was to investigate the efficacy of various uncertainty estimation methods in improving segmentation reliability. We evaluated their confidence levels in voxel predictions and ability to reveal potential segmentation errors.Approach.We retrospectively collected data from 567 HNC patients with diverse cancer sites and multi-modality images (CT, PET, T1-, and T2-weighted MRI) along with their clinical GTV-T/N delineations. Using the nnUNet 3D segmentation pipeline, we compared seven uncertainty estimation methods, evaluating them based on segmentation accuracy (Dice similarity coefficient, DSC), confidence calibration (Expected Calibration Error, ECE), and their ability to reveal segmentation errors (Uncertainty-Error overlap using DSC, UE-DSC).Main results.Evaluated on the hold-out test dataset (n= 97), the median DSC scores for GTV-T and GTV-N segmentation across all uncertainty estimation methods had a narrow range, from 0.73 to 0.76 and 0.78 to 0.80, respectively. In contrast, the median ECE exhibited a wider range, from 0.30 to 0.12 for GTV-T and 0.25 to 0.09 for GTV-N. Similarly, the median UE-DSC also ranged broadly, from 0.21 to 0.38 for GTV-T and 0.22 to 0.36 for GTV-N. A probabilistic network-PhiSeg method consistently demonstrated the best performance in terms of ECE and UE-DSC.Significance.Our study highlights the importance of uncertainty estimation in enhancing the reliability of deep learning for autosegmentation of HNC GTV. The results show that while segmentation accuracy can be similar across methods, their reliability, measured by calibration error and uncertainty-error overlap, varies significantly. Used with visualisation maps, these methods may effectively pinpoint uncertainties and potential errors at the voxel level.
Collapse
Affiliation(s)
- Jintao Ren
- Danish Centre for Particle Therapy, Aarhus University Hospital, Palle Juul-Jensens Boulevard 25, 8200 Aarhus N, Denmark
- Department of Oncology, Aarhus University Hospital, Palle Juul-Jensens Boulevard 25, 8200 Aarhus N, Denmark
- Department of Clinical Medicine, Aarhus University, Palle Juul-Jensens Boulevard 25, 8200 Aarhus N, Denmark
| | - Jonas Teuwen
- Department of Radiation Oncology, Netherlands Cancer Institute, Plesmanlaan 121, 1066 CX Amsterdam, The Netherlands
| | - Jasper Nijkamp
- Danish Centre for Particle Therapy, Aarhus University Hospital, Palle Juul-Jensens Boulevard 25, 8200 Aarhus N, Denmark
- Department of Clinical Medicine, Aarhus University, Palle Juul-Jensens Boulevard 25, 8200 Aarhus N, Denmark
| | - Mathis Rasmussen
- Danish Centre for Particle Therapy, Aarhus University Hospital, Palle Juul-Jensens Boulevard 25, 8200 Aarhus N, Denmark
- Department of Oncology, Aarhus University Hospital, Palle Juul-Jensens Boulevard 25, 8200 Aarhus N, Denmark
- Department of Clinical Medicine, Aarhus University, Palle Juul-Jensens Boulevard 25, 8200 Aarhus N, Denmark
| | - Zeno Gouw
- Department of Radiation Oncology, Netherlands Cancer Institute, Plesmanlaan 121, 1066 CX Amsterdam, The Netherlands
| | - Jesper Grau Eriksen
- Department of Oncology, Aarhus University Hospital, Palle Juul-Jensens Boulevard 25, 8200 Aarhus N, Denmark
- Department of Clinical Medicine, Aarhus University, Palle Juul-Jensens Boulevard 25, 8200 Aarhus N, Denmark
| | - Jan-Jakob Sonke
- Department of Radiation Oncology, Netherlands Cancer Institute, Plesmanlaan 121, 1066 CX Amsterdam, The Netherlands
| | - Stine Korreman
- Danish Centre for Particle Therapy, Aarhus University Hospital, Palle Juul-Jensens Boulevard 25, 8200 Aarhus N, Denmark
- Department of Oncology, Aarhus University Hospital, Palle Juul-Jensens Boulevard 25, 8200 Aarhus N, Denmark
- Department of Clinical Medicine, Aarhus University, Palle Juul-Jensens Boulevard 25, 8200 Aarhus N, Denmark
| |
Collapse
|
2
|
Sahlsten J, Jaskari J, Wahid KA, Ahmed S, Glerean E, He R, Kann BH, Mäkitie A, Fuller CD, Naser MA, Kaski K. Application of simultaneous uncertainty quantification and segmentation for oropharyngeal cancer use-case with Bayesian deep learning. COMMUNICATIONS MEDICINE 2024; 4:110. [PMID: 38851837 PMCID: PMC11162474 DOI: 10.1038/s43856-024-00528-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/19/2023] [Accepted: 05/16/2024] [Indexed: 06/10/2024] Open
Abstract
BACKGROUND Radiotherapy is a core treatment modality for oropharyngeal cancer (OPC), where the primary gross tumor volume (GTVp) is manually segmented with high interobserver variability. This calls for reliable and trustworthy automated tools in clinician workflow. Therefore, accurate uncertainty quantification and its downstream utilization is critical. METHODS Here we propose uncertainty-aware deep learning for OPC GTVp segmentation, and illustrate the utility of uncertainty in multiple applications. We examine two Bayesian deep learning (BDL) models and eight uncertainty measures, and utilize a large multi-institute dataset of 292 PET/CT scans to systematically analyze our approach. RESULTS We show that our uncertainty-based approach accurately predicts the quality of the deep learning segmentation in 86.6% of cases, identifies low performance cases for semi-automated correction, and visualizes regions of the scans where the segmentations likely fail. CONCLUSIONS Our BDL-based analysis provides a first-step towards more widespread implementation of uncertainty quantification in OPC GTVp segmentation.
Collapse
Affiliation(s)
- Jaakko Sahlsten
- Department of Computer Science, Aalto University School of Science, Espoo, Finland
| | - Joel Jaskari
- Department of Computer Science, Aalto University School of Science, Espoo, Finland
| | - Kareem A Wahid
- Department of Radiation Oncology, The University of Texas MD Anderson Cancer Center, Houston, TX, USA
| | - Sara Ahmed
- Department of Radiation Oncology, The University of Texas MD Anderson Cancer Center, Houston, TX, USA
| | - Enrico Glerean
- Department of Neuroscience and Biomedical Engineering, Aalto University School of Science, Espoo, Finland
| | - Renjie He
- Department of Radiation Oncology, The University of Texas MD Anderson Cancer Center, Houston, TX, USA
| | - Benjamin H Kann
- Artificial Intelligence in Medicine Program, Brigham and Women's Hospital, Dana-Farber Cancer Institute, Harvard Medical School, Boston, MA, USA
| | - Antti Mäkitie
- Department of Otorhinolaryngology, Head and Neck Surgery, University of Helsinki and Helsinki University Hospital, Helsinki, Finland
- Research Program in Systems Oncology, University of Helsinki, Helsinki, Finland
| | - Clifton D Fuller
- Department of Radiation Oncology, The University of Texas MD Anderson Cancer Center, Houston, TX, USA
| | - Mohamed A Naser
- Department of Radiation Oncology, The University of Texas MD Anderson Cancer Center, Houston, TX, USA.
| | - Kimmo Kaski
- Department of Computer Science, Aalto University School of Science, Espoo, Finland.
| |
Collapse
|
3
|
Wahid KA, Kaffey ZY, Farris DP, Humbert-Vidan L, Moreno AC, Rasmussen M, Ren J, Naser MA, Netherton TJ, Korreman S, Balakrishnan G, Fuller CD, Fuentes D, Dohopolski MJ. Artificial Intelligence Uncertainty Quantification in Radiotherapy Applications - A Scoping Review. MEDRXIV : THE PREPRINT SERVER FOR HEALTH SCIENCES 2024:2024.05.13.24307226. [PMID: 38798581 PMCID: PMC11118597 DOI: 10.1101/2024.05.13.24307226] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/29/2024]
Abstract
Background/purpose The use of artificial intelligence (AI) in radiotherapy (RT) is expanding rapidly. However, there exists a notable lack of clinician trust in AI models, underscoring the need for effective uncertainty quantification (UQ) methods. The purpose of this study was to scope existing literature related to UQ in RT, identify areas of improvement, and determine future directions. Methods We followed the PRISMA-ScR scoping review reporting guidelines. We utilized the population (human cancer patients), concept (utilization of AI UQ), context (radiotherapy applications) framework to structure our search and screening process. We conducted a systematic search spanning seven databases, supplemented by manual curation, up to January 2024. Our search yielded a total of 8980 articles for initial review. Manuscript screening and data extraction was performed in Covidence. Data extraction categories included general study characteristics, RT characteristics, AI characteristics, and UQ characteristics. Results We identified 56 articles published from 2015-2024. 10 domains of RT applications were represented; most studies evaluated auto-contouring (50%), followed by image-synthesis (13%), and multiple applications simultaneously (11%). 12 disease sites were represented, with head and neck cancer being the most common disease site independent of application space (32%). Imaging data was used in 91% of studies, while only 13% incorporated RT dose information. Most studies focused on failure detection as the main application of UQ (60%), with Monte Carlo dropout being the most commonly implemented UQ method (32%) followed by ensembling (16%). 55% of studies did not share code or datasets. Conclusion Our review revealed a lack of diversity in UQ for RT applications beyond auto-contouring. Moreover, there was a clear need to study additional UQ methods, such as conformal prediction. Our results may incentivize the development of guidelines for reporting and implementation of UQ in RT.
Collapse
Affiliation(s)
- Kareem A. Wahid
- Department of Imaging Physics, The University of Texas MD Anderson Cancer Center, Houston, Texas, USA
- Department of Radiation Oncology, The University of Texas MD Anderson Cancer Center, Houston, Texas, USA
| | - Zaphanlene Y. Kaffey
- Department of Radiation Oncology, The University of Texas MD Anderson Cancer Center, Houston, Texas, USA
| | - David P. Farris
- Research Medical Library, The University of Texas MD Anderson Cancer Center, Houston, Texas, USA
| | - Laia Humbert-Vidan
- Department of Radiation Oncology, The University of Texas MD Anderson Cancer Center, Houston, Texas, USA
| | - Amy C. Moreno
- Department of Radiation Oncology, The University of Texas MD Anderson Cancer Center, Houston, Texas, USA
| | | | - Jintao Ren
- Department of Oncology, Aarhus University Hospital, Denmark
| | - Mohamed A. Naser
- Department of Radiation Oncology, The University of Texas MD Anderson Cancer Center, Houston, Texas, USA
| | - Tucker J. Netherton
- Department of Radiation Physics, University of Texas MD Anderson Cancer Center, Houston, TX, USA
| | - Stine Korreman
- Department of Oncology, Aarhus University Hospital, Denmark
| | | | - Clifton D. Fuller
- Department of Radiation Oncology, The University of Texas MD Anderson Cancer Center, Houston, Texas, USA
| | - David Fuentes
- Department of Imaging Physics, The University of Texas MD Anderson Cancer Center, Houston, Texas, USA
| | - Michael J. Dohopolski
- Department of Radiation Oncology, The University of Texas Southwestern Medical Center, Dallas, Texas, USA
| |
Collapse
|
4
|
Rodríguez Outeiral R, Ferreira Silvério N, González PJ, Schaake EE, Janssen T, van der Heide UA, Simões R. A network score-based metric to optimize the quality assurance of automatic radiotherapy target segmentations. Phys Imaging Radiat Oncol 2023; 28:100500. [PMID: 37869474 PMCID: PMC10587515 DOI: 10.1016/j.phro.2023.100500] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/23/2023] [Revised: 10/06/2023] [Accepted: 10/09/2023] [Indexed: 10/24/2023] Open
Abstract
Background and purpose Existing methods for quality assurance of the radiotherapy auto-segmentations focus on the correlation between the average model entropy and the Dice Similarity Coefficient (DSC) only. We identified a metric directly derived from the output of the network and correlated it with clinically relevant metrics for contour accuracy. Materials and Methods Magnetic Resonance Imaging auto-segmentations were available for the gross tumor volume for cervical cancer brachytherapy (106 segmentations) and for the clinical target volume for rectal cancer external-beam radiotherapy (77 segmentations). The nnU-Net's output before binarization was taken as a score map. We defined a metric as the mean of the voxels in the score map above a threshold (λ). Comparisons were made with the mean and standard deviation over the score map and with the mean over the entropy map. The DSC, the 95th Hausdorff distance, the mean surface distance (MSD) and the surface DSC were computed for segmentation quality. Correlations between the studied metrics and model quality were assessed with the Pearson correlation coefficient (r). The area under the curve (AUC) was determined for detecting segmentations that require reviewing. Results For both tasks, our metric (λ = 0.30) correlated more strongly with the segmentation quality than the mean over the entropy map (for surface DSC, r > 0.65 vs. r < 0.60). The AUC was above 0.84 for detecting MSD values above 2 mm. Conclusions Our metric correlated strongly with clinically relevant segmentation metrics and detected segmentations that required reviewing, indicating its potential for automatic quality assurance of radiotherapy target auto-segmentations.
Collapse
Affiliation(s)
- Roque Rodríguez Outeiral
- Department of Radiation Oncology, The Netherlands Cancer Institute, Plesmanlaan 121, 1066 CX Amsterdam, the Netherlands
| | - Nicole Ferreira Silvério
- Department of Radiation Oncology, The Netherlands Cancer Institute, Plesmanlaan 121, 1066 CX Amsterdam, the Netherlands
| | - Patrick J. González
- Department of Radiation Oncology, The Netherlands Cancer Institute, Plesmanlaan 121, 1066 CX Amsterdam, the Netherlands
| | - Eva E. Schaake
- Department of Radiation Oncology, The Netherlands Cancer Institute, Plesmanlaan 121, 1066 CX Amsterdam, the Netherlands
| | - Tomas Janssen
- Department of Radiation Oncology, The Netherlands Cancer Institute, Plesmanlaan 121, 1066 CX Amsterdam, the Netherlands
| | - Uulke A. van der Heide
- Department of Radiation Oncology, The Netherlands Cancer Institute, Plesmanlaan 121, 1066 CX Amsterdam, the Netherlands
| | - Rita Simões
- Department of Radiation Oncology, The Netherlands Cancer Institute, Plesmanlaan 121, 1066 CX Amsterdam, the Netherlands
| |
Collapse
|
5
|
De Kerf G, Claessens M, Raouassi F, Mercier C, Stas D, Ost P, Dirix P, Verellen D. A geometry and dose-volume based performance monitoring of artificial intelligence models in radiotherapy treatment planning for prostate cancer. Phys Imaging Radiat Oncol 2023; 28:100494. [PMID: 37809056 PMCID: PMC10550805 DOI: 10.1016/j.phro.2023.100494] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/06/2023] [Revised: 09/20/2023] [Accepted: 09/20/2023] [Indexed: 10/10/2023] Open
Abstract
Background and Purpose Clinical Artificial Intelligence (AI) implementations lack ground-truth when applied on real-world data. This study investigated how combined geometrical and dose-volume metrics can be used as performance monitoring tools to detect clinically relevant candidates for model retraining. Materials and Methods Fifty patients were analyzed for both AI-segmentation and planning. For AI-segmentation, geometrical (Standard Surface Dice 3 mm and Local Surface Dice 3 mm) and dose-volume based parameters were calculated for two organs (bladder and anorectum) to compare AI output against the clinically corrected structure. A Local Surface Dice was introduced to detect geometrical changes in the vicinity of the target volumes, while an Absolute Dose Difference (ADD) evaluation increased focus on dose-volume related changes. AI-planning performance was evaluated using clinical goal analysis in combination with volume and target overlap metrics. Results The Local Surface Dice reported equal or lower values compared to the Standard Surface Dice (anorectum: (0.93 ± 0.11) vs (0.98 ± 0.04); bladder: (0.97 ± 0.06) vs (0.98 ± 0.04)). The ADD metric showed a difference of (0.9 ± 0.8)Gy for the anorectum D 1 cm 3 . The bladder D 5cm 3 reported a difference of (0.7 ± 1.5)Gy. Mandatory clinical goals were fulfilled in 90 % of the DLP plans. Conclusions Combining dose-volume and geometrical metrics allowed detection of clinically relevant changes, applied to both auto-segmentation and auto-planning output and the Local Surface Dice was more sensitive to local changes compared to the Standard Surface Dice. This monitoring is able to evaluate AI behavior in clinical practice and allows candidate selection for active learning.
Collapse
Affiliation(s)
- Geert De Kerf
- Department of Radiation Oncology, Iridium Netwerk, Wilrijk (Antwerp), Belgium
| | - Michaël Claessens
- Department of Radiation Oncology, Iridium Netwerk, Wilrijk (Antwerp), Belgium
- Centre for Oncological Research (CORE), Integrated Personalized and Precision Oncology Network (IPPON), University of Antwerp, Antwerp, Belgium
| | - Fadoua Raouassi
- Department of Radiation Oncology, Iridium Netwerk, Wilrijk (Antwerp), Belgium
| | - Carole Mercier
- Department of Radiation Oncology, Iridium Netwerk, Wilrijk (Antwerp), Belgium
- Centre for Oncological Research (CORE), Integrated Personalized and Precision Oncology Network (IPPON), University of Antwerp, Antwerp, Belgium
| | - Daan Stas
- Department of Radiation Oncology, Iridium Netwerk, Wilrijk (Antwerp), Belgium
- Faculty of Medicine and Health Sciences, University of Antwerp, Antwerp, Belgium
| | - Piet Ost
- Department of Radiation Oncology, Iridium Netwerk, Wilrijk (Antwerp), Belgium
- Centre for Oncological Research (CORE), Integrated Personalized and Precision Oncology Network (IPPON), University of Antwerp, Antwerp, Belgium
| | - Piet Dirix
- Department of Radiation Oncology, Iridium Netwerk, Wilrijk (Antwerp), Belgium
- Centre for Oncological Research (CORE), Integrated Personalized and Precision Oncology Network (IPPON), University of Antwerp, Antwerp, Belgium
| | - Dirk Verellen
- Department of Radiation Oncology, Iridium Netwerk, Wilrijk (Antwerp), Belgium
- Centre for Oncological Research (CORE), Integrated Personalized and Precision Oncology Network (IPPON), University of Antwerp, Antwerp, Belgium
| |
Collapse
|
6
|
Gu X, Strijbis VIJ, Slotman BJ, Dahele MR, Verbakel WFAR. Dose distribution prediction for head-and-neck cancer radiotherapy using a generative adversarial network: influence of input data. Front Oncol 2023; 13:1251132. [PMID: 37829347 PMCID: PMC10565853 DOI: 10.3389/fonc.2023.1251132] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/30/2023] [Accepted: 08/25/2023] [Indexed: 10/14/2023] Open
Abstract
Purpose A three-dimensional deep generative adversarial network (GAN) was used to predict dose distributions for locally advanced head and neck cancer radiotherapy. Given the labor- and time-intensive nature of manual planning target volume (PTV) and organ-at-risk (OAR) segmentation, we investigated whether dose distributions could be predicted without the need for fully segmented datasets. Materials and methods GANs were trained/validated/tested using 320/30/35 previously segmented CT datasets and treatment plans. The following input combinations were used to train and test the models: CT-scan only (C); CT+PTVboost/elective (CP); CT+PTVs+OARs+body structure (CPOB); PTVs+OARs+body structure (POB); PTVs+body structure (PB). Mean absolute errors (MAEs) for the predicted dose distribution and mean doses to individual OARs (individual salivary glands, individual swallowing structures) were analyzed. Results For the five models listed, MAEs were 7.3 Gy, 3.5 Gy, 3.4 Gy, 3.4 Gy, and 3.5 Gy, respectively, without significant differences among CP-CPOB, CP-POB, CP-PB, among CPOB-POB. Dose volume histograms showed that all four models that included PTV contours predicted dose distributions that had a high level of agreement with clinical treatment plans. The best model CPOB and the worst model PB (except model C) predicted mean dose to within ±3 Gy of the clinical dose, for 82.6%/88.6%/82.9% and 71.4%/67.1%/72.2% of all OARs, parotid glands (PG), and submandibular glands (SMG), respectively. The R2 values (0.17/0.96/0.97/0.95/0.95) of OAR mean doses for each model also indicated that except for model C, the predictions correlated highly with the clinical dose distributions. Interestingly model C could reasonably predict the dose in eight patients, but on average, it performed inadequately. Conclusion We demonstrated the influence of the CT scan, and PTV and OAR contours on dose prediction. Model CP was not statistically different from model CPOB and represents the minimum data statistically required to adequately predict the clinical dose distribution in a group of patients.
Collapse
Affiliation(s)
- Xiaojin Gu
- Department of Radiation Oncology, Amsterdam UMC Location Vrije Universiteit Amsterdam, Amsterdam, Netherlands
- Cancer Center Amsterdam, Cancer Treatment and Quality of Life, Amsterdam, Netherlands
| | - Victor I. J. Strijbis
- Department of Radiation Oncology, Amsterdam UMC Location Vrije Universiteit Amsterdam, Amsterdam, Netherlands
- Cancer Center Amsterdam, Cancer Treatment and Quality of Life, Amsterdam, Netherlands
| | - Ben J. Slotman
- Department of Radiation Oncology, Amsterdam UMC Location Vrije Universiteit Amsterdam, Amsterdam, Netherlands
- Cancer Center Amsterdam, Cancer Treatment and Quality of Life, Amsterdam, Netherlands
| | - Max R. Dahele
- Department of Radiation Oncology, Amsterdam UMC Location Vrije Universiteit Amsterdam, Amsterdam, Netherlands
- Cancer Center Amsterdam, Cancer Treatment and Quality of Life, Amsterdam, Netherlands
| | - Wilko F. A. R. Verbakel
- Department of Radiation Oncology, Amsterdam UMC Location Vrije Universiteit Amsterdam, Amsterdam, Netherlands
- Cancer Center Amsterdam, Cancer Treatment and Quality of Life, Amsterdam, Netherlands
| |
Collapse
|
7
|
van den Berg I, Savenije MH, Teunissen FR, van de Pol SM, Rasing MJ, van Melick HH, Brink WM, de Boer JC, van den Berg CA, van der Voort van Zyp JR. Deep learning for automated contouring of neurovascular structures on magnetic resonance imaging for prostate cancer patients. Phys Imaging Radiat Oncol 2023; 26:100453. [PMID: 37312973 PMCID: PMC10258498 DOI: 10.1016/j.phro.2023.100453] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/02/2023] [Revised: 05/24/2023] [Accepted: 05/26/2023] [Indexed: 06/15/2023] Open
Abstract
Background and purpose Manual contouring of neurovascular structures on prostate magnetic resonance imaging (MRI) is labor-intensive and prone to considerable interrater disagreement. Our aim is to contour neurovascular structures automatically on prostate MRI by deep learning (DL) to improve workflow and interrater agreement. Materials and methods Segmentation of neurovascular structures was performed on pre-treatment 3.0 T MRI data of 131 prostate cancer patients (training [n = 105] and testing [n = 26]). The neurovascular structures include the penile bulb (PB), corpora cavernosa (CCs), internal pudendal arteries (IPAs), and neurovascular bundles (NVBs). Two DL networks, nnU-Net and DeepMedic, were trained for auto-contouring on prostate MRI and evaluated using volumetric Dice similarity coefficient (DSC), mean surface distances (MSD), Hausdorff distances, and surface DSC. Three radiation oncologists evaluated the DL-generated contours and performed corrections when necessary. Interrater agreement was assessed and the time required for manual correction was recorded. Results nnU-Net achieved a median DSC of 0.92 (IQR: 0.90-0.93) for the PB, 0.90 (IQR: 0.86-0.92) for the CCs, 0.79 (IQR: 0.77-0.83) for the IPAs, and 0.77 (IQR: 0.72-0.81) for the NVBs, which outperformed DeepMedic for each structure (p < 0.03). nnU-Net showed a median MSD of 0.24 mm for the IPAs and 0.71 mm for the NVBs. The median interrater DSC ranged from 0.93 to 1.00, with the majority of cases (68.9%) requiring manual correction times under two minutes. Conclusions DL enables reliable auto-contouring of neurovascular structures on pre-treatment MRI data, easing the clinical workflow in neurovascular-sparing MR-guided radiotherapy.
Collapse
Affiliation(s)
- Ingeborg van den Berg
- Department of Radiation Oncology, Division of Imaging & Oncology, University Medical Center Utrecht, Utrecht, The Netherlands
- Department of Urology, St. Antonius Hospital, Nieuwegein, Utrecht, The Netherlands
- Magnetic Detection and Imaging Group, Technical Medical Centre, University of Twente, Enschede, the Netherlands
| | - Mark H.F. Savenije
- Department of Radiation Oncology, Division of Imaging & Oncology, University Medical Center Utrecht, Utrecht, The Netherlands
| | - Frederik R. Teunissen
- Department of Radiation Oncology, Division of Imaging & Oncology, University Medical Center Utrecht, Utrecht, The Netherlands
| | - Sandrine M.G. van de Pol
- Department of Radiation Oncology, Division of Imaging & Oncology, University Medical Center Utrecht, Utrecht, The Netherlands
| | - Marnix J.A. Rasing
- Department of Radiation Oncology, Division of Imaging & Oncology, University Medical Center Utrecht, Utrecht, The Netherlands
| | - Harm H.E. van Melick
- Department of Urology, St. Antonius Hospital, Nieuwegein, Utrecht, The Netherlands
| | - Wyger M. Brink
- Magnetic Detection and Imaging Group, Technical Medical Centre, University of Twente, Enschede, the Netherlands
| | - Johannes C.J. de Boer
- Department of Radiation Oncology, Division of Imaging & Oncology, University Medical Center Utrecht, Utrecht, The Netherlands
| | - Cornelis A.T. van den Berg
- Department of Radiation Oncology, Division of Imaging & Oncology, University Medical Center Utrecht, Utrecht, The Netherlands
| | | |
Collapse
|
8
|
Sahlsten J, Jaskari J, Wahid KA, Ahmed S, Glerean E, He R, Kann BH, Mäkitie A, Fuller CD, Naser MA, Kaski K. Application of simultaneous uncertainty quantification for image segmentation with probabilistic deep learning: Performance benchmarking of oropharyngeal cancer target delineation as a use-case. MEDRXIV : THE PREPRINT SERVER FOR HEALTH SCIENCES 2023:2023.02.20.23286188. [PMID: 36865296 PMCID: PMC9980236 DOI: 10.1101/2023.02.20.23286188] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 02/26/2023]
Abstract
Background Oropharyngeal cancer (OPC) is a widespread disease, with radiotherapy being a core treatment modality. Manual segmentation of the primary gross tumor volume (GTVp) is currently employed for OPC radiotherapy planning, but is subject to significant interobserver variability. Deep learning (DL) approaches have shown promise in automating GTVp segmentation, but comparative (auto)confidence metrics of these models predictions has not been well-explored. Quantifying instance-specific DL model uncertainty is crucial to improving clinician trust and facilitating broad clinical implementation. Therefore, in this study, probabilistic DL models for GTVp auto-segmentation were developed using large-scale PET/CT datasets, and various uncertainty auto-estimation methods were systematically investigated and benchmarked. Methods We utilized the publicly available 2021 HECKTOR Challenge training dataset with 224 co-registered PET/CT scans of OPC patients with corresponding GTVp segmentations as a development set. A separate set of 67 co-registered PET/CT scans of OPC patients with corresponding GTVp segmentations was used for external validation. Two approximate Bayesian deep learning methods, the MC Dropout Ensemble and Deep Ensemble, both with five submodels, were evaluated for GTVp segmentation and uncertainty performance. The segmentation performance was evaluated using the volumetric Dice similarity coefficient (DSC), mean surface distance (MSD), and Hausdorff distance at 95% (95HD). The uncertainty was evaluated using four measures from literature: coefficient of variation (CV), structure expected entropy, structure predictive entropy, and structure mutual information, and additionally with our novel Dice-risk measure. The utility of uncertainty information was evaluated with the accuracy of uncertainty-based segmentation performance prediction using the Accuracy vs Uncertainty (AvU) metric, and by examining the linear correlation between uncertainty estimates and DSC. In addition, batch-based and instance-based referral processes were examined, where the patients with high uncertainty were rejected from the set. In the batch referral process, the area under the referral curve with DSC (R-DSC AUC) was used for evaluation, whereas in the instance referral process, the DSC at various uncertainty thresholds were examined. Results Both models behaved similarly in terms of the segmentation performance and uncertainty estimation. Specifically, the MC Dropout Ensemble had 0.776 DSC, 1.703 mm MSD, and 5.385 mm 95HD. The Deep Ensemble had 0.767 DSC, 1.717 mm MSD, and 5.477 mm 95HD. The uncertainty measure with the highest DSC correlation was structure predictive entropy with correlation coefficients of 0.699 and 0.692 for the MC Dropout Ensemble and the Deep Ensemble, respectively. The highest AvU value was 0.866 for both models. The best performing uncertainty measure for both models was the CV which had R-DSC AUC of 0.783 and 0.782 for the MC Dropout Ensemble and Deep Ensemble, respectively. With referring patients based on uncertainty thresholds from 0.85 validation DSC for all uncertainty measures, on average the DSC improved from the full dataset by 4.7% and 5.0% while referring 21.8% and 22% patients for MC Dropout Ensemble and Deep Ensemble, respectively. Conclusion We found that many of the investigated methods provide overall similar but distinct utility in terms of predicting segmentation quality and referral performance. These findings are a critical first-step towards more widespread implementation of uncertainty quantification in OPC GTVp segmentation.
Collapse
Affiliation(s)
- Jaakko Sahlsten
- Department of Computer Science, Aalto University School of Science, Espoo, Finland
| | - Joel Jaskari
- Department of Computer Science, Aalto University School of Science, Espoo, Finland
| | - Kareem A Wahid
- Department of Radiation Oncology, The University of Texas MD Anderson Cancer Center, Houston, TX USA
| | - Sara Ahmed
- Department of Radiation Oncology, The University of Texas MD Anderson Cancer Center, Houston, TX USA
| | - Enrico Glerean
- Department of Neuroscience and Biomedical Engineering, Aalto University School of Science, Espoo, Finland
| | - Renjie He
- Department of Radiation Oncology, The University of Texas MD Anderson Cancer Center, Houston, TX USA
| | - Benjamin H Kann
- Artificial Intelligence in Medicine Program, Brigham and Women's Hospital, Dana-Farber Cancer Institute, Harvard Medical School, Boston, MA USA
| | - Antti Mäkitie
- Department of Otorhinolaryngology, Head and Neck Surgery, University of Helsinki and Helsinki University Hospital, Helsinki, Finland
| | - Clifton D Fuller
- Department of Radiation Oncology, The University of Texas MD Anderson Cancer Center, Houston, TX USA
| | - Mohamed A Naser
- Department of Radiation Oncology, The University of Texas MD Anderson Cancer Center, Houston, TX USA
| | - Kimmo Kaski
- Department of Computer Science, Aalto University School of Science, Espoo, Finland
| |
Collapse
|
9
|
Strijbis VIJ, Dahele M, Gurney-Champion OJ, Blom GJ, Vergeer MR, Slotman BJ, Verbakel WFAR. Deep Learning for Automated Elective Lymph Node Level Segmentation for Head and Neck Cancer Radiotherapy. Cancers (Basel) 2022; 14:cancers14225501. [PMID: 36428593 PMCID: PMC9688342 DOI: 10.3390/cancers14225501] [Citation(s) in RCA: 6] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/30/2022] [Revised: 10/31/2022] [Accepted: 11/07/2022] [Indexed: 11/11/2022] Open
Abstract
Depending on the clinical situation, different combinations of lymph node (LN) levels define the elective LN target volume in head-and-neck cancer (HNC) radiotherapy. The accurate auto-contouring of individual LN levels could reduce the burden and variability of manual segmentation and be used regardless of the primary tumor location. We evaluated three deep learning approaches for the segmenting individual LN levels I−V, which were manually contoured on CT scans from 70 HNC patients. The networks were trained and evaluated using five-fold cross-validation and ensemble learning for 60 patients with (1) 3D patch-based UNets, (2) multi-view (MV) voxel classification networks and (3) sequential UNet+MV. The performances were evaluated using Dice similarity coefficients (DSC) for automated and manual segmentations for individual levels, and the planning target volumes were extrapolated from the combined levels I−V and II−IV, both for the cross-validation and for an independent test set of 10 patients. The median DSC were 0.80, 0.66 and 0.82 for UNet, MV and UNet+MV, respectively. Overall, UNet+MV significantly (p < 0.0001) outperformed other arrangements and yielded DSC = 0.87, 0.85, 0.86, 0.82, 0.77, 0.77 for the combined and individual level I−V structures, respectively. Both PTVs were also significantly (p < 0.0001) more accurate with UNet+MV, with DSC = 0.91 and 0.90, respectively. The accurate segmentation of individual LN levels I−V can be achieved using an ensemble of UNets. UNet+MV can further refine this result.
Collapse
Affiliation(s)
- Victor I. J. Strijbis
- Department of Radiation Oncology, Amsterdam UMC Location Vrije Universiteit Amsterdam, De Boelelaan 1117, 1081 HV Amsterdam, The Netherlands
- Cancer Center Amsterdam, Cancer Treatment and Quality of Life, 1081 HV Amsterdam, The Netherlands
- Correspondence: ; Tel.: +31-6-54-32-64-03
| | - Max Dahele
- Department of Radiation Oncology, Amsterdam UMC Location Vrije Universiteit Amsterdam, De Boelelaan 1117, 1081 HV Amsterdam, The Netherlands
- Cancer Center Amsterdam, Cancer Treatment and Quality of Life, 1081 HV Amsterdam, The Netherlands
| | - Oliver J. Gurney-Champion
- Department of Radiology and Nuclear Medicine, Amsterdam UMC Location University of Amsterdam, Meibergdreef 9, 1105 AZ Amsterdam, The Netherlands
- Cancer Center Amsterdam, Imaging and Biomarkers, 1081 HV Amsterdam, The Netherlands
| | - Gerrit J. Blom
- Department of Radiation Oncology, Amsterdam UMC Location Vrije Universiteit Amsterdam, De Boelelaan 1117, 1081 HV Amsterdam, The Netherlands
| | - Marije R. Vergeer
- Department of Radiation Oncology, Amsterdam UMC Location Vrije Universiteit Amsterdam, De Boelelaan 1117, 1081 HV Amsterdam, The Netherlands
| | - Berend J. Slotman
- Department of Radiation Oncology, Amsterdam UMC Location Vrije Universiteit Amsterdam, De Boelelaan 1117, 1081 HV Amsterdam, The Netherlands
- Cancer Center Amsterdam, Cancer Treatment and Quality of Life, 1081 HV Amsterdam, The Netherlands
| | - Wilko F. A. R. Verbakel
- Department of Radiation Oncology, Amsterdam UMC Location Vrije Universiteit Amsterdam, De Boelelaan 1117, 1081 HV Amsterdam, The Netherlands
- Cancer Center Amsterdam, Cancer Treatment and Quality of Life, 1081 HV Amsterdam, The Netherlands
| |
Collapse
|
10
|
Delpon G, Barateau A, Beneux A, Bessières I, Latorzeff I, Welmant J, Tallet A. [What do we need to deliver "online" adapted radiotherapy treatment plans?]. Cancer Radiother 2022; 26:794-802. [PMID: 36028418 DOI: 10.1016/j.canrad.2022.06.024] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/22/2022] [Revised: 06/27/2022] [Accepted: 06/29/2022] [Indexed: 11/17/2022]
Abstract
During the joint SFRO/SFPM session of the 2019 congress, a state of the art of adaptive radiotherapy announced a strong impact in our clinical practice, in particular with the availability of treatment devices coupled to an MRI system. Three years later, it seems relevant to take stock of adaptive radiotherapy in practice, and especially the "online" strategy because it is indeed more and more accessible with recent hardware and software developments, such as coupled accelerators to a three-dimensional imaging device and algorithms based on artificial intelligence. However, the deployment of this promising strategy is complex because it contracts the usual time scale and upsets the usual organizations. So what do we need to deliver adapted treatment plans with an "online" strategy?
Collapse
Affiliation(s)
- G Delpon
- Institut de cancérologie de l'Ouest, Saint-Herblain et IMT Atlantique, Nantes université, CNRS/IN2P3, Subatech, Nantes, France.
| | - A Barateau
- Université Rennes, CLCC Eugène-Marquis, Inserm, LTSI-UMR 1099, Rennes, France
| | - A Beneux
- Hospices Civils de Lyon, Lyon, France
| | - I Bessières
- Centre Georges-François Leclerc, Dijon, France
| | | | - J Welmant
- Institut du cancer de Montpellier, Montpellier, France
| | - A Tallet
- Institut Paoli-Calmettes, Marseille, France
| |
Collapse
|
11
|
Claessens M, Vanreusel V, De Kerf G, Mollaert I, Löfman F, Gooding MJ, Brouwer C, Dirix P, Verellen D. Machine learning-based detection of aberrant deep learning segmentations of target and organs at risk for prostate radiotherapy using a secondary segmentation algorithm. Phys Med Biol 2022; 67. [PMID: 35561701 DOI: 10.1088/1361-6560/ac6fad] [Citation(s) in RCA: 11] [Impact Index Per Article: 5.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/18/2021] [Accepted: 05/13/2022] [Indexed: 11/11/2022]
Abstract
Objective.The output of a deep learning (DL) auto-segmentation application should be reviewed, corrected if needed and approved before being used clinically. This verification procedure is labour-intensive, time-consuming and user-dependent, which potentially leads to significant errors with impact on the overall treatment quality. Additionally, when the time needed to correct auto-segmentations approaches the time to delineate target and organs at risk from scratch, the usability of the DL model can be questioned. Therefore, an automated quality assurance framework was developed with the aim to detect in advance aberrant auto-segmentations.Approach. Five organs (prostate, bladder, anorectum, femoral head left and right) were auto-delineated on CT acquisitions for 48 prostate patients by an in-house trained primary DL model. An experienced radiation oncologist assessed the correctness of the model output and categorised the auto-segmentations into two classes whether minor or major adaptations were needed. Subsequently, an independent, secondary DL model was implemented to delineate the same structures as the primary model. Quantitative comparison metrics were calculated using both models' segmentations and used as input features for a machine learning classification model to predict the output quality of the primary model.Main results. For every organ, the approach of independent validation by the secondary model was able to detect primary auto-segmentations that needed major adaptation with high sensitivity (recall = 1) based on the calculated quantitative metrics. The surface DSC and APL were found to be the most indicated parameters in comparison to standard quantitative metrics for the time needed to adapt auto-segmentations.Significance. This proposed method includes a proof of concept for the use of an independent DL segmentation model in combination with a ML classifier to improve time saving during QA of auto-segmentations. The integration of such system into current automatic segmentation pipelines can increase the efficiency of the radiotherapy contouring workflow.
Collapse
Affiliation(s)
- Michaël Claessens
- Department of Radiation Oncology, Iridium Network, Wilrijk (Antwerp), Belgium.,Centre for Oncological Research (CORE), Integrated Personalized and Precision Oncology Network (IPPON), University of Antwerp, Belgium
| | - Verdi Vanreusel
- Department of Radiation Oncology, Iridium Network, Wilrijk (Antwerp), Belgium
| | - Geert De Kerf
- Department of Radiation Oncology, Iridium Network, Wilrijk (Antwerp), Belgium
| | - Isabelle Mollaert
- Department of Radiation Oncology, Iridium Network, Wilrijk (Antwerp), Belgium
| | - Fredrik Löfman
- Department of Machine Learning, RaySearch Laboratories AB, Stockholm, Sweden
| | | | - Charlotte Brouwer
- University of Groningen, University Medical Center Groningen, Department of Radiation Oncology, The Netherlands
| | - Piet Dirix
- Department of Radiation Oncology, Iridium Network, Wilrijk (Antwerp), Belgium.,Centre for Oncological Research (CORE), Integrated Personalized and Precision Oncology Network (IPPON), University of Antwerp, Belgium
| | - Dirk Verellen
- Department of Radiation Oncology, Iridium Network, Wilrijk (Antwerp), Belgium.,Centre for Oncological Research (CORE), Integrated Personalized and Precision Oncology Network (IPPON), University of Antwerp, Belgium
| |
Collapse
|
12
|
Recent Applications of Artificial Intelligence in Radiotherapy: Where We Are and Beyond. APPLIED SCIENCES-BASEL 2022. [DOI: 10.3390/app12073223] [Citation(s) in RCA: 10] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/12/2022]
Abstract
In recent decades, artificial intelligence (AI) tools have been applied in many medical fields, opening the possibility of finding novel solutions for managing very complex and multifactorial problems, such as those commonly encountered in radiotherapy (RT). We conducted a PubMed and Scopus search to identify the AI application field in RT limited to the last four years. In total, 1824 original papers were identified, and 921 were analyzed by considering the phase of the RT workflow according to the applied AI approaches. AI permits the processing of large quantities of information, data, and images stored in RT oncology information systems, a process that is not manageable for individuals or groups. AI allows the iterative application of complex tasks in large datasets (e.g., delineating normal tissues or finding optimal planning solutions) and might support the entire community working in the various sectors of RT, as summarized in this overview. AI-based tools are now on the roadmap for RT and have been applied to the entire workflow, mainly for segmentation, the generation of synthetic images, and outcome prediction. Several concerns were raised, including the need for harmonization while overcoming ethical, legal, and skill barriers.
Collapse
|
13
|
Lappas G, Wolfs CJA, Staut N, Lieuwes NG, Biemans R, van Hoof SJ, Dubois LJ, Verhaegen F. Automatic contouring of normal tissues with deep learning for preclinical radiation studies. Phys Med Biol 2022; 67. [PMID: 35061600 DOI: 10.1088/1361-6560/ac4da3] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/22/2021] [Accepted: 01/21/2022] [Indexed: 02/05/2023]
Abstract
Objective.Delineation of relevant normal tissues is a bottleneck in image-guided precision radiotherapy workflows for small animals. A deep learning (DL) model for automatic contouring using standardized 3D micro cone-beam CT (μCBCT) volumes as input is proposed, to provide a fully automatic, generalizable method for normal tissue contouring in preclinical studies.Approach.A 3D U-net was trained to contour organs in the head (whole brain, left/right brain hemisphere, left/right eye) and thorax (complete lungs, left/right lung, heart, spinal cord, thorax bone) regions. As an important preprocessing step, Hounsfield units (HUs) were converted to mass density (MD) values, to remove the energy dependency of theμCBCT scanner and improve generalizability of the DL model. Model performance was evaluated quantitatively by Dice similarity coefficient (DSC), mean surface distance (MSD), 95th percentile Hausdorff distance (HD95p), and center of mass displacement (ΔCoM). For qualitative assessment, DL-generated contours (for 40 and 80 kV images) were scored (0: unacceptable, manual re-contouring needed - 5: no adjustments needed). An uncertainty analysis using Monte Carlo dropout uncertainty was performed for delineation of the heart.Main results.The proposed DL model and accompanying preprocessing method provide high quality contours, with in general median DSC > 0.85, MSD < 0.25 mm, HD95p < 1 mm and ΔCoM < 0.5 mm. The qualitative assessment showed very few contours needed manual adaptations (40 kV: 20/155 contours, 80 kV: 3/155 contours). The uncertainty of the DL model is small (within 2%).Significance.A DL-based model dedicated to preclinical studies has been developed for multi-organ segmentation in two body sites. For the first time, a method independent of image acquisition parameters has been quantitatively evaluated, resulting in sub-millimeter performance, while qualitative assessment demonstrated the high quality of the DL-generated contours. The uncertainty analysis additionally showed that inherent model variability is low.
Collapse
Affiliation(s)
- Georgios Lappas
- Department of Radiation Oncology (Maastro), GROW-School for Oncology and Developmental Biology, Maastricht University Medical Centre+, Maastricht, The Netherlands
| | - Cecile J A Wolfs
- Department of Radiation Oncology (Maastro), GROW-School for Oncology and Developmental Biology, Maastricht University Medical Centre+, Maastricht, The Netherlands
| | - Nick Staut
- Department of Radiation Oncology (Maastro), GROW-School for Oncology and Developmental Biology, Maastricht University Medical Centre+, Maastricht, The Netherlands.,SmART Scientific Solutions BV, Maastricht, The Netherlands
| | - Natasja G Lieuwes
- The M-Lab, Department of Precision Medicine, GROW-School for Oncology and Developmental Biology, Maastricht University, Maastricht, The Netherlands
| | - Rianne Biemans
- The M-Lab, Department of Precision Medicine, GROW-School for Oncology and Developmental Biology, Maastricht University, Maastricht, The Netherlands
| | | | - Ludwig J Dubois
- The M-Lab, Department of Precision Medicine, GROW-School for Oncology and Developmental Biology, Maastricht University, Maastricht, The Netherlands
| | - Frank Verhaegen
- Department of Radiation Oncology (Maastro), GROW-School for Oncology and Developmental Biology, Maastricht University Medical Centre+, Maastricht, The Netherlands.,SmART Scientific Solutions BV, Maastricht, The Netherlands
| |
Collapse
|