Analysis of Geometric Performance and Dosimetric Impact of Using Automatic Contour Segmentation for Radiotherapy Planning

Sensitivity of automated and manual treatment planning approaches to contouring variation in early-breast cancer treatment

PURPOSE

One of the advantages of integrating automated processes in treatment planning is the reduction of manual planning variability. This study aims to assess whether a deep-learning-based auto-planning solution can also reduce the contouring variation-related impact on the planned dose for early-breast cancer treatment.

METHODS

Auto- and manual plans were optimized for 20 patients using both auto- and manual OARs, including both lungs, right breast, heart, and left-anterior-descending (LAD) artery. Differences in terms of recalculated dose (ΔDrcM,ΔDrcA) and reoptimized dose (ΔDroM,ΔDroA) for manual (M) and auto (A)-plans, were evaluated on manual structures. The correlation between several geometric similarities and dose differences was also explored (Spearman's test).

RESULTS

Auto-contours were found slightly smaller in size than manual contours for right breast and heart and more than twice larger for LAD. Recalculated dose differences were found negligible for both planning approaches except for heart (ΔDrcM=-0.4 Gy, ΔDrcA=-0.3 Gy) and right breast (ΔDrcM=-1.2 Gy, ΔDrcA=-1.3 Gy) maximum dose. Re-optimized dose differences were considered equivalent to recalculated ones for both lungs and LAD, while they were significantly smaller for heart (ΔDroM=-0.2 Gy, ΔDroA=-0.2 Gy) and right breast (ΔDroM =-0.3 Gy, ΔDroA=-0.9 Gy) maximum dose. Twenty-one correlations were found for ΔDrcM,A (M=8,A=13) that reduced to four for ΔDroM,A (M=3,A=1).

CONCLUSIONS

The sensitivity of auto-planning to contouring variation was found not relevant when compared to manual planning, regardless of the method used to calculate the dose differences. Nonetheless, the method employed to define the dose differences strongly affected the correlation analysis resulting highly reduced when dose was reoptimized, regardless of the planning approach.

Essentially unedited deep-learning-based OARs are suitable for rigorous oropharyngeal and laryngeal cancer treatment planning

Quality of organ at risk (OAR) autosegmentation is often judged by concordance metrics against the human-generated gold standard. However, the ultimate goal is the ability to use unedited autosegmented OARs in treatment planning, while maintaining the plan quality. We tested this approach with head and neck (HN) OARs generated by a prototype deep-learning (DL) model on patients previously treated for oropharyngeal and laryngeal cancer. Forty patients were selected, with all structures delineated by an experienced physician. For each patient, a set of 13 OARs were generated by the DL model. Each patient was re-planned based on original targets and unedited DL-produced OARs. The new dose distributions were then applied back to the manually delineated structures. The target coverage was evaluated with inhomogeneity index (II) and the relative volume of regret. For the OARs, Dice similarity coefficient (DSC) of areas under the DVH curves, individual DVH objectives, and composite continuous plan quality metric (PQM) were compared. The nearly identical primary target coverage for the original and re-generated plans was achieved, with the same II and relative volume of regret values. The average DSC of the areas under the corresponding pairs of DVH curves was 0.97 ± 0.06. The number of critical DVH points which met the clinical objectives with the dose optimized on autosegmented structures but failed when evaluated on the manual ones was 5 of 896 (0.6%). The average OAR PQM score with the re-planned dose distributions was essentially the same when evaluated either on the autosegmented or manual OARs. Thus, rigorous HN treatment planning is possible with OARs segmented by a prototype DL algorithm with minimal, if any, manual editing.

Clinical Use of a Commercial Artificial Intelligence-Based Software for Autocontouring in Radiation Therapy: Geometric Performance and Dosimetric Impact

PURPOSE

When autocontouring based on artificial intelligence (AI) is used in the radiotherapy (RT) workflow, the contours are reviewed and eventually adjusted by a radiation oncologist before an RT treatment plan is generated, with the purpose of improving dosimetry and reducing both interobserver variability and time for contouring. The purpose of this study was to evaluate the results of application of a commercial AI-based autocontouring for RT, assessing both geometric accuracies and the influence on optimized dose from automatically generated contours after review by human operator.

MATERIALS AND METHODS

A commercial autocontouring system was applied to a retrospective database of 40 patients, of which 20 were treated with radiotherapy for prostate cancer (PCa) and 20 for head and neck cancer (HNC). Contours resulting from AI were compared against AI contours reviewed by human operator and human-only contours using Dice similarity coefficient (DSC), Hausdorff distance (HD), and relative volume difference (RVD). Dosimetric indices such as Dmean, D0.03cc, and normalized plan quality metrics were used to compare dose distributions from RT plans generated from structure sets contoured by humans assisted by AI against plans from manual contours. The reduction in contouring time obtained by using automated tools was also assessed. A Wilcoxon rank sum test was computed to assess the significance of differences. Interobserver variability of the comparison of manual vs. AI-assisted contours was also assessed among two radiation oncologists for PCa.

RESULTS

For PCa, AI-assisted segmentation showed good agreement with expert radiation oncologist structures with average DSC among patients ≥ 0.7 for all structures, and minimal radiation oncology adjustment of structures (DSC of adjusted versus AI structures ≥ 0.91). For HNC, results of comparison between manual and AI contouring varied considerably e.g., 0.77 for oral cavity and 0.11-0.13 for brachial plexus, but again, adjustment was generally minimal (DSC of adjusted against AI contours 0.97 for oral cavity, 0.92-0.93 for brachial plexus). The difference in dose for the target and organs at risk were not statistically significant between human and AI-assisted, with the only exceptions of D0.03cc to the anal canal and Dmean to the brachial plexus. The observed average differences in plan quality for PCa and HNC cases were 8% and 6.7%, respectively. The dose parameter changes due to interobserver variability in PCa were small, with the exception of the anal canal, where large dose variations were observed. The reduction in time required for contouring was 72% for PCa and 84% for HNC.

CONCLUSIONS

When an autocontouring system is used in combination with human review, the time of the RT workflow is significantly reduced without affecting dose distribution and plan quality.

Dosimetric comparison of autocontouring techniques for online adaptive proton therapy

Objective.Anatomical and daily set-up uncertainties impede high precision delivery of proton therapy. With online adaptation, the daily plan is reoptimized on an image taken shortly before the treatment, reducing these uncertainties and, hence, allowing a more accurate delivery. This reoptimization requires target and organs-at-risk (OAR) contours on the daily image, which need to be delineated automatically since manual contouring is too slow. Whereas multiple methods for autocontouring exist, none of them are fully accurate, which affects the daily dose. This work aims to quantify the magnitude of this dosimetric effect for four contouring techniques.Approach.Plans reoptimized on automatic contours are compared with plans reoptimized on manual contours. The methods include rigid and deformable registration (DIR), deep-learning based segmentation and patient-specific segmentation.Main results.It was found that independently of the contouring method, the dosimetric influence of usingautomaticOARcontoursis small (<5% prescribed dose in most cases), with DIR yielding the best results. Contrarily, the dosimetric effect of using theautomatic target contourwas larger (>5% prescribed dose in most cases), indicating that manual verification of that contour remains necessary. However, when compared to non-adaptive therapy, the dose differences caused by automatically contouring the target were small and target coverage was improved, especially for DIR.Significance.The results show that manual adjustment of OARs is rarely necessary and that several autocontouring techniques are directly usable. Contrarily, manual adjustment of the target is important. This allows prioritizing tasks during time-critical online adaptive proton therapy and therefore supports its further clinical implementation.

Automated Contouring and Planning in Radiation Therapy: What Is 'Clinically Acceptable'?

Developers and users of artificial-intelligence-based tools for automatic contouring and treatment planning in radiotherapy are expected to assess clinical acceptability of these tools. However, what is 'clinical acceptability'? Quantitative and qualitative approaches have been used to assess this ill-defined concept, all of which have advantages and disadvantages or limitations. The approach chosen may depend on the goal of the study as well as on available resources. In this paper, we discuss various aspects of 'clinical acceptability' and how they can move us toward a standard for defining clinical acceptability of new autocontouring and planning tools.

Accuracy and Feasibility of Synthetic CT for Lung Adaptive Radiotherapy: A Phantom Study

OBJECTIVES

The respiratory variations will lead to inconsistency between the actual delivery dose and the planning dose. How the minor interfractional amplitude changes affect the geometry and dose delivery accuracy remains to be investigated in the context of lung adaptive radiotherapy.

METHODS

Planning 4-dimensional-computed tomography and kV-cone beam computed tomography were scanned based on the Computerized Imaging Reference Systems phantom, which was employed to simulate the minor interfractional amplitude variations. The corresponding synthetic computed tomography for a particular motion pattern can be generated from Velocity program. Then a clinically meaningful synthetic computed tomography was analyzed through the geometrical and dosimetric assessment.

RESULTS

The image quality of synthetic computed tomography was improved obviously compared with cone beam computed tomography. Mean absolute error was minimized when no significant interfractional motion occurs and Velocity can be qualified for dealing with the regular breathing motion patterns. The mean percent hounsfield unit difference of the synthetic hounsfield unit values per organ relative to the planning 4-dimensional-computed tomography image was 22.3%. Under the same conditions, the mean percent hounsfield unit difference of the cone beam computed tomography hounsfield unit values per organ, relative to the planning 4-dimensional-computed tomography image was 83.9%. Overall, the accuracy of hounsfield unit in synthetic computed tomography was improved obviously and the variability of the synthetic image correlates with the planning 4-dimensional-computed tomography image variability. Meanwhile, the dose-volume histograms between planning 4-dimensional-computed tomography and synthetic computed tomography almost coincided each other, which indicates that Velocity program can qualify lung adaptive radiotherapy well when there were no interfractional respiratory variations. However, for cases with obvious interfractional amplitude change, the volume covered at least by 100% of the prescription dose was only 59.6% for that synthetic image.

CONCLUSION

The synthetic computed tomography images generated from Velocity were close to the real images in anatomy and dosimetry, which can make clinical lung adaptive radiotherapy possible based on the actual patient anatomy during treatment.

Clinical validation of deep learning algorithms for radiotherapy targeting of non-small-cell lung cancer: an observational study

BACKGROUND

Artificial intelligence (AI) and deep learning have shown great potential in streamlining clinical tasks. However, most studies remain confined to in silico validation in small internal cohorts, without external validation or data on real-world clinical utility. We developed a strategy for the clinical validation of deep learning models for segmenting primary non-small-cell lung cancer (NSCLC) tumours and involved lymph nodes in CT images, which is a time-intensive step in radiation treatment planning, with large variability among experts.

METHODS

In this observational study, CT images and segmentations were collected from eight internal and external sources from the USA, the Netherlands, Canada, and China, with patients from the Maastro and Harvard-RT1 datasets used for model discovery (segmented by a single expert). Validation consisted of interobserver and intraobserver benchmarking, primary validation, functional validation, and end-user testing on the following datasets: multi-delineation, Harvard-RT1, Harvard-RT2, RTOG-0617, NSCLC-radiogenomics, Lung-PET-CT-Dx, RIDER, and thorax phantom. Primary validation consisted of stepwise testing on increasingly external datasets using measures of overlap including volumetric dice (VD) and surface dice (SD). Functional validation explored dosimetric effect, model failure modes, test-retest stability, and accuracy. End-user testing with eight experts assessed automated segmentations in a simulated clinical setting.

FINDINGS

We included 2208 patients imaged between 2001 and 2015, with 787 patients used for model discovery and 1421 for model validation, including 28 patients for end-user testing. Models showed an improvement over the interobserver benchmark (multi-delineation dataset; VD 0·91 [IQR 0·83-0·92], p=0·0062; SD 0·86 [0·71-0·91], p=0·0005), and were within the intraobserver benchmark. For primary validation, AI performance on internal Harvard-RT1 data (segmented by the same expert who segmented the discovery data) was VD 0·83 (IQR 0·76-0·88) and SD 0·79 (0·68-0·88), within the interobserver benchmark. Performance on internal Harvard-RT2 data segmented by other experts was VD 0·70 (0·56-0·80) and SD 0·50 (0·34-0·71). Performance on RTOG-0617 clinical trial data was VD 0·71 (0·60-0·81) and SD 0·47 (0·35-0·59), with similar results on diagnostic radiology datasets NSCLC-radiogenomics and Lung-PET-CT-Dx. Despite these geometric overlap results, models yielded target volumes with equivalent radiation dose coverage to those of experts. We also found non-significant differences between de novo expert and AI-assisted segmentations. AI assistance led to a 65% reduction in segmentation time (5·4 min; p<0·0001) and a 32% reduction in interobserver variability (SD; p=0·013).

INTERPRETATION

We present a clinical validation strategy for AI models. We found that in silico geometric segmentation metrics might not correlate with clinical utility of the models. Experts' segmentation style and preference might affect model performance.

FUNDING

US National Institutes of Health and EU European Research Council.

Explaining the dosimetric impact of contouring errors in head and neck radiotherapy

Objective. Auto-contouring of organs at risk (OAR) is becoming more common in radiotherapy. An important issue in clinical decision making is judging the quality of the auto-contours. While recent studies considered contour quality by looking at geometric errors only, this does not capture the dosimetric impact of the errors. In this work, we studied the relationship between geometrical errors, the local dose and the dosimetric impact of the geometrical errors.Approach. For 94 head and neck patients, unmodified atlas-based auto-contours and clinically used delineations of the parotid glands and brainstem were retrieved. VMAT plans were automatically optimized on the auto-contours and evaluated on both contours. We defined the dosimetric impact on evaluation (DIE) as the difference in the dosimetric parameter of interest between the two contours. We developed three linear regression models to predict the DIE using: (1) global geometric metrics, (2) global dosimetric metrics, (3) combined local geometric and dosimetric metrics. For model (3), we next determined the minimal amount of editing information required to produce a reliable prediction. Performance was assessed by the root mean squared error (RMSE) of the predicted DIE using 5-fold cross-validation.Main results. In model (3), the median RMSE of the left parotid was 0.4 Gy using 5% of the largest editing vectors. For the right parotid and brainstem the results were 0.5 Gy using 10% and 0.4 Gy using 1% respectively. The median RMS of the DIE was 0.6 Gy, 0.7 Gy and 0.9 Gy for the left parotid, the right parotid and the brainstem, respectively. Model (3), combining local dosimetric and geometric quantities, outperformed the models that used only geometric or dosimetric information.Significance. We showed that the largest local errors plus the local dose suffice to accurately predict the dosimetric impact, opening the door to automated dosimetric QA of auto-contours.

Automatic contouring QA method using a deep learning-based autocontouring system

Purpose

To determine the most accurate similarity metric when using an independent system to verify automatically generated contours.

Methods

A reference autocontouring system (primary system to create clinical contours) and a verification autocontouring system (secondary system to test the primary contours) were used to generate a pair of 6 female pelvic structures (UteroCervix [uterus + cervix], CTVn [nodal clinical target volume (CTV)], PAN [para‐aortic lymph nodes], bladder, rectum, and kidneys) on 49 CT scans from our institution and 38 from other institutions. Additionally, clinically acceptable and unacceptable contours were manually generated using the 49 internal CT scans. Eleven similarity metrics (volumetric Dice similarity coefficient (DSC), Hausdorff distance, 95% Hausdorff distance, mean surface distance, and surface DSC with tolerances from 1 to 10 mm) were calculated between the reference and the verification autocontours, and between the manually generated and the verification autocontours. A support vector machine (SVM) was used to determine the threshold that separates clinically acceptable and unacceptable contours for each structure. The 11 metrics were investigated individually and in certain combinations. Linear, radial basis function, sigmoid, and polynomial kernels were tested using the combinations of metrics as inputs for the SVM.

Results

The highest contouring error detection accuracies were 0.91 for the UteroCervix, 0.90 for the CTVn, 0.89 for the PAN, 0.92 for the bladder, 0.95 for the rectum, and 0.97 for the kidneys and were achieved using surface DSCs with a thickness of 1, 2, or 3 mm. The linear kernel was the most accurate and consistent when a combination of metrics was used as an input for the SVM. However, the best model accuracy from the combinations of metrics was not better than the best model accuracy from a surface DSC as an input.

Conclusions

We distinguished clinically acceptable contours from clinically unacceptable contours with an accuracy higher than 0.9 for the targets and critical structures in patients with cervical cancer; the most accurate similarity metric was surface DSC with a thickness of 1, 2, or 3 mm.

Dosimetric Study of Deep Learning-Guided ITV Prediction in Cone-beam CT for Lung Stereotactic Body Radiotherapy

Purpose

The purpose of this study was to evaluate the accuracy of a lung stereotactic body radiotherapy (SBRT) treatment plan with the target of a newly predicted internal target volume (ITV_predict) and the feasibility of its clinical application. ITV_predict was automatically generated by our in-house deep learning model according to the cone-beam CT (CBCT) image database.

Method

A retrospective study of 45 patients who underwent SBRT was involved, and Mask R-CNN based algorithm model helped to predict the internal target volume (ITV) using the CBCT image database. The geometric accuracy of ITV_predict was verified by the Dice Similarity Coefficient (DSC), 3D Motion Range (R_3D), Relative Volume Index (RVI), and Hausdorff Distance (HD). The PTV_predict was generated by ITV_predict, which was registered and then projected on free-breath CT (FBCT) images. The PTV_FBCT was margined from the GTV on FBCT images gross tumor volume on free-breath CT (GTV_FBCT). Treatment plans with the target of Predict planning target volume on CBCT images (PTV_predict) and planning target volume on free-breath CT (PTV_FBCT) were respectively re-established, and the dosimetric parameters included the ratio of the volume of patients receiving at least the prescribed dose to the volume of PTV (R_100%), the ratio of the volume of patients receiving at least 50% of the prescribed dose to the volume of PTV in the Radiation Therapy Oncology Group (RTOG) 0813 Trial (R_50%), Gradient Index (GI), and the maximum dose 2 cm from the PTV (D_2cm), which were evaluated via Plan_4DCT, plan which based on PTV_predict (Plan_predict), and plan which based on PTV_FBCT (Plan_FBCT).

Result

The geometric results showed that there existed a good correlation between ITV_predict and ITV on the 4-dimensional CT [ITV_4DCT; DSC= 0.83 ±0.18]. However, the average volume of ITV_predict was 10% less than that of ITV_4DCT (p = 0.333). No significant difference in dose coverage was found in V_100% for the ITV with 99.98 ± 0.04% in the ITV_4DCT vs. 97.56 ± 4.71% in the ITV_predict (p = 0.162). Dosimetry parameters of PTV, including R_100%, R_50%, GI and D_2cm showed no statistically significant difference between each plan (p > 0.05).

Conclusion

Dosimetric parameters of Plan_predict are clinically comparable to those of the original Plan_4DCT. This study confirmed that the treatment plan based on ITV_predict produced by our model could automatically meet clinical requirements. Thus, for patients undergoing lung SBRT, the model has great potential for using CBCT images for ITV contouring which can be used in treatment planning.

Machine Learning for Auto-Segmentation in Radiotherapy Planning

Manual segmentation of target structures and organs at risk is a crucial step in the radiotherapy workflow. It has the disadvantages that it can require several hours of clinician time per patient and is prone to inter- and intra-observer variability. Automatic segmentation (auto-segmentation), using computer algorithms, seeks to address these issues. Advances in machine learning and computer vision have led to the development of methods for accurate and efficient auto-segmentation. This review surveys auto-segmentation techniques and applications in radiotherapy planning. It provides an overview of traditional approaches to auto-segmentation, including intensity analysis, shape modelling and atlas-based methods. The focus, though, is on uses of machine learning and deep learning, including convolutional neural networks. Finally, the future of machine-learning-driven auto-segmentation in clinical settings is considered, and the barriers that must be overcome for it to be widely accepted into routine practice are highlighted.

Geometric and Dosimetric Evaluation of Deep Learning-Based Automatic Delineation on CBCT-Synthesized CT and Planning CT for Breast Cancer Adaptive Radiotherapy: A Multi-Institutional Study

Purpose

We developed a deep learning model to achieve automatic multitarget delineation on planning CT (pCT) and synthetic CT (sCT) images generated from cone-beam CT (CBCT) images. The geometric and dosimetric impact of the model was evaluated for breast cancer adaptive radiation therapy.

Methods

We retrospectively analyzed 1,127 patients treated with radiotherapy after breast-conserving surgery from two medical institutions. The CBCT images for patient setup acquired utilizing breath-hold guided by optical surface monitoring system were used to generate sCT with a generative adversarial network. Organs at risk (OARs), clinical target volume (CTV), and tumor bed (TB) were delineated automatically with a 3D U-Net model on pCT and sCT images. The geometric accuracy of the model was evaluated with metrics, including Dice similarity coefficient (DSC) and 95% Hausdorff distance (HD95). Dosimetric evaluation was performed by quick dose recalculation on sCT images relying on gamma analysis and dose-volume histogram (DVH) parameters. The relationship between ΔD95, ΔV95 and DSC-CTV was assessed to quantify the clinical impact of the geometric changes of CTV.

Results

The ranges of DSC and HD95 were 0.73–0.97 and 2.22–9.36 mm for pCT, 0.63–0.95 and 2.30–19.57 mm for sCT from institution A, 0.70–0.97 and 2.10–11.43 mm for pCT from institution B, respectively. The quality of sCT was excellent with an average mean absolute error (MAE) of 71.58 ± 8.78 HU. The mean gamma pass rate (3%/3 mm criterion) was 91.46 ± 4.63%. DSC-CTV down to 0.65 accounted for a variation of more than 6% of V95 and 3 Gy of D95. DSC-CTV up to 0.80 accounted for a variation of less than 4% of V95 and 2 Gy of D95. The mean ΔD90/ΔD95 of CTV and TB were less than 2Gy/4Gy, 4Gy/5Gy for all the patients. The cardiac dose difference in left breast cancer cases was larger than that in right breast cancer cases.

Conclusions

The accurate multitarget delineation is achievable on pCT and sCT via deep learning. The results show that dose distribution needs to be considered to evaluate the clinical impact of geometric variations during breast cancer radiotherapy.

The impact of organ-at-risk contour variations on automatically generated treatment plans for NSCLC

BACKGROUND AND PURPOSE

Quality of automatic contouring is generally assessed by comparison with manual delineations, but the effect of contour differences on the resulting dose distribution remains unknown. This study evaluated dosimetric differences between treatment plans optimized using various organ-at-risk (OAR) contouring methods.

MATERIALS AND METHODS

OARs of twenty lung cancer patients were manually and automatically contoured, after which user-adjustments were made. For each contour set, an automated treatment plan was generated. The dosimetric effect of intra-observer contour variation and the influence of contour variations on treatment plan evaluation and generation were studied using dose-volume histogram (DVH)-parameters for thoracic OARs.

RESULTS

Dosimetric effect of intra-observer contour variability was highest for Heart D_max (3.4 ± 6.8 Gy) and lowest for Lungs-GTV D_mean (0.3 ± 0.4 Gy). The effect of contour variation on treatment plan evaluation was highest for Heart D_max (6.0 ± 13.4 Gy) and Esophagus D_max (8.7 ± 17.2 Gy). Dose differences for the various treatment plans, evaluated on the reference (manual) contour, were on average below 1 Gy/1%. For Heart D_mean, higher dose differences were found for overlap with PTV (median 0.2 Gy, 95% 1.7 Gy) vs. no PTV overlap (median 0 Gy, 95% 0.5 Gy). For D_max-parameters, largest dose difference was found between 0-1 cm distance to PTV (median 1.5 Gy, 95% 4.7 Gy).

CONCLUSION

Dose differences arising from automatic contour variations were of the same magnitude or lower than intra-observer contour variability. For Heart D_mean, we recommend delineation errors to be corrected when the heart overlaps with the PTV. For D_max-parameters, we recommend checking contours if the distance is close to PTV (<5 cm). For the lungs, only obvious large errors need to be adjusted.