Discrepancies in volume calculations between different radiotherapy treatment planning systems

Clinical impact of DVH uncertainties

Dose-volume histograms (DVH), along with dose and volume metrics, are central to radiotherapy planning. As such, errors have the potential to significantly impact the selection of appropriate treatment plans. Dose distributions that pass tests in one TPS may fail the same tests when transferred to another, even if using identical structures and dose grid information. This work shows the design and implementation of methods for assessing the accuracy of dose and volume computations performed by treatment planning systems (TPS), and other analytical tools. We demonstrate examples where differences in calculations between systems can change the assessment of a plan's clinical acceptability. Our work also provides a more detailed DVH analysis of single targets than earlier published studies. This is relevant for SRS plans and small structure dose assessments. Very small structures are a particular problem because of their coarse digital representation, and the impact of this is thoroughly examined. Reference DVH curves were derived mathematically, based on Gaussian dose distributions centered on spherical structures. The structures and dose distributions were generated synthetically, and imported into RayStation, MasterPlan, and ProKnow. Corresponding DVHs were analytically derived and taken as ground truth references, for comparison with the commercial DVH calculations. Two commonly used dose metrics PCI and MGI were used to determine the limit of calculation accuracy for small structures. In addition, to measure the DVH differences between a larger range of commercial DVH calculators, the D95 metric from a set of real clinical plans was compared across both the 3 DVH calculators under test, and across a further six TPSs from other hospitals. We show that even slight deviations between the results of DVH calculators can lead to plan check failures, and we illustrate this with the commonly used D95 planning metric. We present clinical data across eight planning systems that highlight instances where plan checks would pass in one software and fail in another due to DVH calculation differences. For the smallest volumes tested, errors of up to 20% were observed in the DVHs. RayStation was tested down to a 3 mm radius sphere (≈0.1 cc) and this showed close to 10% error, reducing to 1% for 10 mm radius (≈4.0 cc) and 0.1% for 20 mm radius (≈33 cc). In clinical plans, the variation in D95 was up to 9% for the smallest volumes, and typically around 2% in the range 0.5 cc-20 cc, and 1% in 20 cc-70 cc, falling to <0.1% for large volumes. Paddick Conformity Index (PCI) and Modified Gradient Index (MGI) are commonly used plan quality indicators for very small volumes. For volumes ≈0.1 cc we observed errors of up to 40% in PCI, and up to 75% in MGI. Our study extends the range of tested DVH calculators in published work, and shows their performance over a wider range of volume sizes. We provide quantitative evidence of the critical need to test the accuracy of DVH calculators in the TPS before clinical use. This work is particularly relevant for both stereotactic plan evaluation and for assessment of small volume doses in published dose constraint recommendations. We demonstrate that significant errors can occur in DVHs for volumes less than 1 cc, even if the volumes themselves are calculated accurately. Even for large structures, deviations between the outputs of DVH calculators can lead to indicated or reported plan check failures if they do not include appropriate tolerances. We urge caution in the use of DVH metrics for these very small volumes and recommend that appropriate DVH uncertainty tolerances are set in organ dose constraints when using them to evaluate clinical plans.

Multicenter Multivendor Evaluation of Dose Volume Histogram Creation Consistencies for 8 Commercial Radiation Therapy Dosimetric Systems

PURPOSE

To evaluate dose volume histogram (DVH) construction differences across 8 major commercial treatment planning systems (TPS) and dose reporting systems for clinically treated plans of various anatomic sites and target sizes.

METHODS AND MATERIALS

Dose files from 10 selected clinically treated plans with a hypofractionation, stereotactic radiation therapy prescription or sharp dose gradients such as head and neck plans ranging from prescription doses of 18 Gy in 1 fraction to 70 Gy in 35 fractions, each calculated at 0.25 and 0.125 cm grid size, were created and anonymized in Eclipse TPS, and exported to 7 other major TPS (Pinnacle, RayStation, and Elements) and dose reporting systems (MIM, Mobius, ProKnow, and Velocity) systems for comparison. Dose-volume constraint points of clinical importance for each plan were collected from each evaluated system (D0.03 cc [Gy], volume, and the mean dose were used for structures without specified constraints). Each reported constraint type and structure volume was normalized to the value from Eclipse for a pairwise comparison. A Wilcoxon rank-sum test was used for statistical significance and a multivariable regression model was evaluated adjusting for plan, grid size, and distance to target center.

RESULTS

For all DVH points relative to Eclipse, all systems reported median values within 1.0% difference of each other; however, they were all different from Eclipse. Considering mean values, Pinnacle, RayStation, and Elements averaged at 1.038, 1.046, and 1.024, respectively, while MIM, Mobius, ProKnow, and Velocity reported 1.026, 1.050, 1.033, and 1.022, respectively relative to Eclipse. Smaller dose grid size improved agreement between the systems marginally without statistical significance. For structure volumes relative to Eclipse, larger differences are seen across all systems with a range in median values up to 3.0% difference and mean up to 10.1% difference.

CONCLUSIONS

Large variations were observed between all systems. Eclipse generally reported, at statistically significant levels, lower values than all other evaluated systems. The nonsignificant change resulting from lowering the dose grid resolution indicates that this resolution may be less important than other aspects of calculating DVH curves, such as the 3-dimensional modeling of the structure.

Single isocenter dynamic conformal arcs-based radiosurgery for brain metastases: Dosimetric comparison with Cyberknife and clinical investigation

Purpose

To compare the dosimetric quality of automatic multiple brain metastases planning (MBM) with that of Cyberknife (CK) based on the clinical tumor condition, such as the tumor number, size, and location.

Methods

76 treatment plans for 46 patients treated with CK were recalculated with the MBM treatment planning system. Conformity index (CI), homogeneity index (HI), gradient index (GI), lesion underdosage volume factor (LUF), healthy tissue overdose volume factor (HTOF), geometric conformity index (g) and mean dose to normal organs were compared between CK and MBM for tumor number, size, shape and distance from the brainstem or chiasm.

Results

The results showed that the mean brain dose was significantly smaller in MBM than CK. CI did not differ between MBM and CK; however, HI was significantly more ideal in CK (p = 0.000), and GI was significantly smaller in MBM (P = 0.000). LUF was larger in CK (p = 0.000) and HTOF and g was larger in MBM (p = 0.003, and 0.012). For single metastases, CK had significantly better HTOF (p = 0.000) and g (p = 0.002), but there were no differences for multiple tumors. Brain dose in MBM was significantly lower and CI was higher for tumors < 30 mm (p = 0.000 and 0.000), whereas HTOF and g for tumors < 10 mm were significantly smaller in CK (p = 0.041 and p = 0.016). Among oval tumors, brain dose, GI and LUF were smaller in MBM, but HTOF and g were smaller in CK. There were no particular trends for tumors close to the brainstem, but HTOF tended to be smaller in CK (0.03 vs. 0.29, p = 0.068) for tumors inside the brainstem.

Conclusions

MBM can reduce the brain dose while achieving a dose distribution quality equivalent to that with CK.

Assessment of Dose-Volume Histogram Precision for Five Clinical Systems

PURPOSE

To investigate the dependency of dose-volume histogram behavior (DVH) and precision on underlying DICOM discretization using shapes and dose distributions with known analytical DVHs for five commercial DVH calculators.

METHODS

DVHs and summary metrics were extracted from all five systems using synthetic DICOM cone and cylinder objects for which the true volume and DVH curves were known. Trends in the curves and metrics were explored by varying the underlying voxelization of the CT image, structure set, and dose grid as well by varying the geometry of the structure and direction of a linear dose gradient. Using synthetic structures allowed for comparison with ground-truth DVH curves to assess their accuracy while an algorithm was additionally developed to assess the precision of each system. The precision was calculated with a novel algorithm that treats any "stair step" behavior in a DVH curve as an uncertainty band and calculates the width, characterized as a percent difference, of the band for various DVH metrics. The underlying voxelization was additionally changed and DVHs were extracted for two clinical examples. The details of how each system calculated DVHs were also investigated and tendencies in the calculated curves, metrics, and precision were related to choices made in the calculation methodology.

RESULTS

Calculation methodology differences that had a noticeable impact on the DVH curves and summary metrics include supersampling beyond the input grids and interpretation of the superior and inferior ends of the structures. Amongst the systems studied, the median precision ranged from 0.902% to 3.22%, and interquartile ranges varied from 1.09% to 3.91%.

CONCLUSIONS

Commercial dose-evaluation solutions can calculate different DVH curves, structure volume measures, and dose statistics for the same input data due to differences in their calculation methodologies. This study highlights the importance of understanding and investigating the DVH calculation when considering a new clinical system and when using more than one system for data transfer. This article is protected by copyright. All rights reserved.

Treatment planning system commissioning of the first clinical biology‐guided radiotherapy machine

Purpose

Methods

Results

Conclusions

Evaluation of the impact of teaching on delineation variation during a virtual stereotactic ablative radiotherapy contouring workshop

Introduction:

Variation in delineation of target volumes/organs at risk (OARs) is well recognised in radiotherapy and may be reduced by several methods including teaching. We evaluated the impact of teaching on contouring variation for thoracic/pelvic stereotactic ablative radiotherapy (SABR) during a virtual contouring workshop.

Materials and methods:

Target volume/OAR contours produced by workshop participants for three cases were evaluated against reference contours using DICE similarity coefficient (DSC) and line domain error (LDE) metrics. Pre- and post-workshop DSC results were compared using Wilcoxon signed ranks test to determine the impact of teaching during the workshop.

Results:

Of 50 workshop participants, paired pre- and post-workshop contours were available for 21 (42%), 20 (40%) and 22 (44%) participants for primary lung cancer, pelvic bone metastasis and pelvic node metastasis cases, respectively. Statistically significant improvements post-workshop in median DSC and LDE results were observed for 6 (50%) and 7 (58%) of 12 structures, respectively, although the magnitude of DSC/LDE improvement was modest in most cases. An increase in median DSC post-workshop ≥0·05 was only observed for GTVbone, IGTVlung and SacralPlex, and reduction in median LDE > 1 mm was only observed for GTVbone, CTVbone and SacralPlex. Post-workshop, median DSC values were >0·7 for 75% of structures. For 92% of the structures, post-workshop contours were considered to be acceptable or within acceptable variation following review by the workshop faculty.

Conclusions:

This study has demonstrated that virtual SABR contouring training is feasible and was associated with some improvements in contouring variation for multiple target volumes/OARs.

Dependence of volume calculation and margin growth accuracy on treatment planning systems for stereotactic radiosurgery

OBJECTIVE

Uncertainties in radiotherapy target structures are partly dependent on differences between volume calculation and margin growing methods in treatment planning systems (TPS). These uncertainties are exacerbated with very small structures such as those common in stereotactic radiosurgery.

METHODS

Data from a national commissioning programme for SRS was used to assess variation in reported volumes for six benchmark cases, including malignant and benign indications. Reported volumes were compared both with and without any margins added according to local practice.

RESULTS

137 plans were submitted, with a total of 311 structures and covering seven TPS. For volumes < 1 cm³ agreement was within 0.05 cm³, and for volumes > 1 cm³ agreement was within 5%. Systematic differences were seen between TPS, partly because of different methods for calculating the end slice volume. About one third of structures had a margin added, of 1-2 mm. Most TPS over-grew the volumes, compared to the approximation of a perfect sphere, especially Pinnacle and Eclipse.

CONCLUSION

Differences between volume calculation methods may lead to 5-10% variation in reported volumes from different TPS. This should be taken into account when comparing multicentre studies, and it is recommended that a minimum volume of 0.05 cm³ be used for any near-point doses to allow more consistent comparisons. When margins are added to small structures, there may be up to 40% difference to nominal margin size. Such differences are still small compared to interobserver variation in delineation. Advances in knowledge: This study quantifies the potential uncertainties in clinical volume calculation and margin growth with small radiosurgical targets.

Methods, software and datasets to verify DVH calculations against analytical values: Twenty years late(r)

PURPOSE

The authors designed data, methods, and metrics that can serve as a standard, independent of any software package, to evaluate dose-volume histogram (DVH) calculation accuracy and detect limitations. The authors use simple geometrical objects at different orientations combined with dose grids of varying spatial resolution with linear 1D dose gradients; when combined, ground truth DVH curves can be calculated analytically in closed form to serve as the absolute standards.

METHODS

dicom RT structure sets containing a small sphere, cylinder, and cone were created programmatically with axial plane spacing varying from 0.2 to 3 mm. Cylinders and cones were modeled in two different orientations with respect to the IEC 1217 Y axis. The contours were designed to stringently but methodically test voxelation methods required for DVH. Synthetic RT dose files were generated with 1D linear dose gradient and with grid resolution varying from 0.4 to 3 mm. Two commercial DVH algorithms-pinnacle (Philips Radiation Oncology Systems) and PlanIQ (Sun Nuclear Corp.)-were tested against analytical values using custom, noncommercial analysis software. In Test 1, axial contour spacing was constant at 0.2 mm while dose grid resolution varied. In Tests 2 and 3, the dose grid resolution was matched to varying subsampled axial contours with spacing of 1, 2, and 3 mm, and difference analysis and metrics were employed: (1) histograms of the accuracy of various DVH parameters (total volume, Dmax, Dmin, and doses to % volume: D99, D95, D5, D1, D0.03 cm(3)) and (2) volume errors extracted along the DVH curves were generated and summarized in tabular and graphical forms.

RESULTS

In Test 1, pinnacle produced 52 deviations (15%) while PlanIQ produced 5 (1.5%). In Test 2, pinnacle and PlanIQ differed from analytical by >3% in 93 (36%) and 18 (7%) times, respectively. Excluding Dmin and Dmax as least clinically relevant would result in 32 (15%) vs 5 (2%) scored deviations for pinnacle vs PlanIQ in Test 1, while Test 2 would yield 53 (25%) vs 17 (8%). In Test 3, statistical analyses of volume errors extracted continuously along the curves show pinnacle to have more errors and higher variability (relative to PlanIQ), primarily due to pinnacle's lack of sufficient 3D grid supersampling. Another major driver for pinnacle errors is an inconsistency in implementation of the "end-capping"; the additional volume resulting from expanding superior and inferior contours halfway to the next slice is included in the total volume calculation, but dose voxels in this expanded volume are excluded from the DVH. PlanIQ had fewer deviations, and most were associated with a rotated cylinder modeled by rectangular axial contours; for coarser axial spacing, the limited number of cross-sectional rectangles hinders the ability to render the true structure volume.

CONCLUSIONS

The method is applicable to any DVH-calculating software capable of importing dicom RT structure set and dose objects (the authors' examples are available for download). It includes a collection of tests that probe the design of the DVH algorithm, measure its accuracy, and identify failure modes. Merits and applicability of each test are discussed.

Modeling and dosimetric performance evaluation of the RayStation treatment planning system

The physics modeling, dose calculation accuracy and plan quality assessment of the RayStation (v3.5) treatment planning system (TPS) is presented in this study, with appropriate comparisons to the more established Pinnacle (v9.2) TPS. Modeling and validation for the Elekta MLCi and Agility beam models resulted in a good match to treatment machine-measured data based on tolerances of 3% for in-field and out-of-field regions, 10% for buildup and penumbral regions, and a gamma 2%/2mm dose/distance acceptance criteria. TPS commissioning using a wide range of appropriately selected dosimetry equipment, and following published guidelines, established the MLC modeling and dose calculation accuracy to be within standard tolerances for all tests performed. In both homogeneous and heterogeneous mediums, central axis calculations agreed with measurements within 2% for open fields and 3% for wedged fields, and within 4% off-axis. Treatment plan comparisons for identical clinical goals were made to Pinnacle for the following complex clinical cases: hypofractionated non-small cell lung carcinoma, head and neck, stereotactic spine, as well as for several standard clinical cases comprising of prostate, brain, and breast plans. DVHs, target, and critical organ doses, as well as measured point doses and gamma indices, applying both local and global (Van Dyk) normalization at 2%/2 mm and 3%/3 mm (10% lower threshold) acceptance criteria for these composite plans were assessed. In addition 3DVH was used to compare the perturbed dose distributions to the TPS 3D dose distributions. For all 32 cases, the patients QA checks showed &gt; 95% of pixels passing 3% global/3mm gamma.

Contouring variations and the role of atlas in non-small cell lung cancer radiation therapy: Analysis of a multi-institutional preclinical trial planning study

PURPOSE

To quantify variations in target and normal structure contouring and evaluate dosimetric impact of these variations in non-small cell lung cancer (NSCLC) cases. To study whether providing an atlas can reduce potential variation.

METHODS AND MATERIALS

Three NSCLC cases were distributed sequentially to multiple institutions for contouring and radiation therapy planning. No segmentation atlas was provided for the first 2 cases (Case 1 and Case 2). Contours were collected from submitted plans and consensus contour sets were generated. The volume variation among institution contours and the deviation of them from consensus contours were analyzed. The dose-volume histograms for individual institution plans were recalculated using consensus contours to quantify the dosimetric changes. An atlas containing targets and critical structures was constructed and was made available when the third case (Case 3) was distributed for planning. The contouring variability in the submitted plans of Case 3 was compared with that in first 2 cases.

RESULTS

Planning target volume (PTV) showed large variation among institutions. The PTV coverage in institutions' plans decreased dramatically when reevaluated using the consensus PTV contour. The PTV contouring consistency did not show improvement with atlas use in Case 3. For normal structures, lung contours presented very good agreement, while the brachial plexus showed the largest variation. The consistency of esophagus and heart contouring improved significantly (t test; P < .05) in Case 3. Major factors contributing to the contouring variation were identified through a survey questionnaire.

CONCLUSIONS

The amount of contouring variations in NSCLC cases was presented. Its impact on dosimetric parameters can be significant. The segmentation atlas improved the contour agreement for esophagus and heart, but not for the PTV in this study. Quality assurance of contouring is essential for a successful multi-institutional clinical trial.

A review of methods of analysis in contouring studies for radiation oncology

Inter-observer variability in anatomical contouring is the biggest contributor to uncertainty in radiation treatment planning. Contouring studies are frequently performed to investigate the differences between multiple contours on common datasets. There is, however, no widely accepted method for contour comparisons. The purpose of this study is to review the literature on contouring studies in the context of radiation oncology, with particular consideration of the contouring comparison methods they employ. A literature search, not limited by date, was conducted using Medline and Google Scholar with key words: contour, variation, delineation, inter/intra observer, uncertainty and trial dummy-run. This review includes a description of the contouring processes and contour comparison metrics used. The use of different processes and metrics according to tumour site and other factors were also investigated with limitations described. A total of 69 relevant studies were identified. The most common tumour sites were prostate (26), lung (10), head and neck cancers (8) and breast (7).The most common metric of comparison was volume used 59 times, followed by dimension and shape used 36 times, and centre of volume used 19 times. Of all 69 publications, 67 used a combination of metrics and two used only one metric for comparison. No clear relationships between tumour site or any other factors that may influence the contouring process and the metrics used to compare contours were observed from the literature. Further studies are needed to assess the advantages and disadvantages of each metric in various situations.

Comparison of simulator-CT versus simulator fluoroscopy versus surface marking based radiation treatment planning: A prospective study by three-dimensional evaluation

BACKGROUND AND PURPOSE

Field placement for Radiation Treatment Planning can be done based on the surface markings or simulator fluoroscopy or simulator with CT facilities. A prospective study was carried out to compare these three techniques of radiation treatment planning to quantitatively find out the difference in normal tissue dosages and target volume coverage in the three groups after three-dimensional evaluation.

PATIENTS AND METHODS

The CT scans of 30 patients in the treatment position, taken on a Shimadzu SCT-3000 TF scanner at 1cm intervals, were transferred to Theraplan-500 three-dimensional radiation treatment planning computer. The normal tissues and target volumes (GTV and CTV) were outlined on all the CT slices as per (ICRU) Report no. 50. Three types of radiation treatment planning was done sequentially: Plan I-based on the surface markings alone, Plan II-based on simulator-fluoroscopy, and Plan III-based on Simulator-CT.

RESULTS

The mean dose to 95% of the clinical target volume (D95) was increased by 4.4 and 6.4% by Plans II and III as compared with Plan I. The mean dose to 3/3rd (D(3/3)) to all the critical organs was decreased by 6.6 and 8.4% by Plans II and III as compared to Plan I. The mean time, in simulator room, for field placement for Plans I-III was 6.2, 14.6 and 44 min, respectively.

CONCLUSIONS

Thus for adequate coverage of target volumes and sparing normal tissues, Simulator-CT based radiation treatment planning is the best method of radiation treatment planning though it is more time consuming.

Volumetric Uncertainty in Radiotherapy

The technologies available to identify anatomical structures (including radiotherapy target and normal tissue 'volumes'), and to deliver dose accurately to these volumes, have improved significantly in the past decade. However, the ability of clinicians to identify volumes accurately and consistently in patients still suffers from uncertainties that arise from human error, inadequate training, lack of consensus on the derivation of volumes and inadequate characterisation of the accuracy and specificity of imaging technologies. Inadequate volume definition of a target can result in treatment failure and, consequently, disease progression; excessive volume may also lead to unnecessary patient injury. This is a serious problem in routine clinical care. In the context of large multi-centre clinical trials, uncertainty and inconsistency in tissue-volume reporting will be carried through to the analysis of treatment effect on outcome, which will subsequently influence the treatment of future patients. Strategies need to be set in place to ensure that the abilities and consistency of clinicians in defining volumes are aligned with the ability of new technologies to present volumetric information. This review seeks to define the concept of volumetric uncertainty and propose a conceptual model that has these errors evaluated and responded to separately. Specifically, we will explore the major causes, consequences of, and possible remediation of volumetric uncertainty, from the point of view of a multidisciplinary radiotherapy clinical environment.