Clinical validation of a graphical method for radiation therapy plan quality assessment

A Visualization and Radiation Treatment Plan Quality Scoring Method for Triage in a Population-Based Context

Purpose

Our purpose was to develop a clinically intuitive and easily understandable scoring method using statistical metrics to visually determine the quality of a radiation treatment plan.

Methods and Materials

Data from 111 patients with head and neck cancer were used to establish a percentile-based scoring system for treatment plan quality evaluation on both a plan-by-plan and objective-by-objective basis. The percentile scores for each clinical objective and the overall treatment plan score were then visualized using a daisy plot. To validate our scoring method, 6 physicians were recruited to assess 60 plans, each using a scoring table consisting of a 5-point Likert scale (with scores ≥3 considered passing). Spearman correlation analysis was conducted to assess the association between increasing treatment plan percentile rank and physician rating, with Likert scores of 1 and 2 representing clinically unacceptable plans, scores of 3 and 4 representing plans needing minor edits, and a score of 5 representing clinically acceptable plans. Receiver operating characteristic curve analysis was used to assess the scoring system's ability to quantify plan quality.

Results

Of the 60 plans scored by the physicians, 8 were deemed as clinically acceptable; these plans had an 89.0th ± 14.5 percentile value using our scoring system. The plans needing minor edits or deemed unacceptable had more variation, with scores falling in the 62.6nd ± 25.1 percentile and 35.6th ± 25.7 percentile, respectively. The estimated Spearman correlation coefficient between the physician score and treatment plan percentile was 0.53 (P < .001), indicating a moderate but statistically significant correlation. Receiver operating characteristic curve analysis demonstrated discernment between acceptable and unacceptable plan quality, with an area under the curve of 0.76.

Conclusions

Our scoring system correlates with physician ratings while providing intuitive visual feedback for identifying good treatment plan quality, thereby indicating its utility in the quality assurance process.

Deep Learning-Based Dose Prediction for Automated, Individualized Quality Assurance of Head and Neck Radiation Therapy Plans

PURPOSE

This study aimed to use deep learning-based dose prediction to assess head and neck (HN) plan quality and identify suboptimal plans.

METHODS AND MATERIALS

A total of 245 volumetric modulated arc therapy HN plans were created using RapidPlan knowledge-based planning (KBP). A subset of 112 high-quality plans was selected under the supervision of an HN radiation oncologist. We trained a 3D Dense Dilated U-Net architecture to predict 3-dimensional dose distributions using 3-fold cross-validation on 90 plans. Model inputs included computed tomography images, target prescriptions, and contours for targets and organs at risk (OARs). The model's performance was assessed on the remaining 22 test plans. We then tested the application of the dose prediction model for automated review of plan quality. Dose distributions were predicted on 14 clinical plans. The predicted versus clinical OAR dose metrics were compared to flag OARs with suboptimal normal tissue sparing using a 2 Gy dose difference or 3% dose-volume threshold. OAR flags were compared with manual flags by 3 HN radiation oncologists.

RESULTS

The predicted dose distributions were of comparable quality to the KBP plans. The differences between the predicted and KBP-planned D1%,D95%, and D99% across the targets were within -2.53% ± 1.34%, -0.42% ± 1.27%, and -0.12% ± 1.97%, respectively, and the OAR mean and maximum doses were within -0.33 ± 1.40 Gy and -0.96 ± 2.08 Gy, respectively. For the plan quality assessment study, radiation oncologists flagged 47 OARs for possible plan improvement. There was high interphysician variability; 83% of physician-flagged OARs were flagged by only one of 3 physicians. The comparative dose prediction model flagged 63 OARs, including 30 of 47 physician-flagged OARs.

CONCLUSIONS

Deep learning can predict high-quality dose distributions, which can be used as comparative dose distributions for automated, individualized assessment of HN plan quality.

Automated Contouring and Planning in Radiation Therapy: What Is 'Clinically Acceptable'?

Developers and users of artificial-intelligence-based tools for automatic contouring and treatment planning in radiotherapy are expected to assess clinical acceptability of these tools. However, what is 'clinical acceptability'? Quantitative and qualitative approaches have been used to assess this ill-defined concept, all of which have advantages and disadvantages or limitations. The approach chosen may depend on the goal of the study as well as on available resources. In this paper, we discuss various aspects of 'clinical acceptability' and how they can move us toward a standard for defining clinical acceptability of new autocontouring and planning tools.