Interobserver variation in parotid gland delineation: a study of its impact on intensity-modulated radiotherapy solutions with a systematic review of the literature

Impact of a contouring atlas on radiographer inter-observer variation in male pelvis radiotherapy

PURPOSE/OBJECTIVE

To determine the impact of a MR-based contouring atlas for male pelvis radiotherapy delineation on inter-observer variation to support radiographer led real-time magnetic resonance image guided adaptive radiotherapy (MRgART).

MATERIAL/METHODS

Eight RTTs contoured 25 MR images in the Monaco treatment planning system (Monaco 5.40.01), from 5 patients. The prostate, seminal vesicles, bladder, and rectum were delineated before and after the introduction of an atlas developed through multi-disciplinary consensus. Inter-observer contour variations (volume), time to contour and observer contouring confidence were determined at both time-points using a 5-point Likert scale. Descriptive statistics were used to analyse both continuous and categorical variables. Dice similarity coefficient (DSC), Dice-Jaccard coefficient (DJC) and Hausdorff distance were used to calculate similarity between observers.

RESULTS

Although variation in volume definition decreased for all structures among all observers post intervention, the change was not statistically significant. DSC and DJC measurements remained consistent following the introduction of the atlas for all observers. The highest similarity was found in the bladder and prostate whilst the lowest was the seminal vesicles. The mean contouring time for all observers was reduced by 50% following the introduction of the atlas (53 to 27 minutes, p=0.01). For all structures across all observers, the mean contouring confidence increased significantly from 2.3 to 3.5 out of 5 (p=0.02).

CONCLUSION

Although no significant improvements were observed in contour variation amongst observers, the introduction of the consensus-based contouring atlas improved contouring confidence and speed; key factors for a real-time RTT-led MRgART.

Reference standard for the evaluation of automatic segmentation algorithms: Quantification of inter observer variability of manual delineation of prostate contour on MRI

PURPOSE

The purpose of this study was to investigate the relationship between inter-reader variability in manual prostate contour segmentation on magnetic resonance imaging (MRI) examinations and determine the optimal number of readers required to establish a reliable reference standard.

MATERIALS AND METHODS

Seven radiologists with various experiences independently performed manual segmentation of the prostate contour (whole-gland [WG] and transition zone [TZ]) on 40 prostate MRI examinations obtained in 40 patients. Inter-reader variability in prostate contour delineations was estimated using standard metrics (Dice similarity coefficient [DSC], Hausdorff distance and volume-based metrics). The impact of the number of readers (from two to seven) on segmentation variability was assessed using pairwise metrics (consistency) and metrics with respect to a reference segmentation (conformity), obtained either with majority voting or simultaneous truth and performance level estimation (STAPLE) algorithm.

RESULTS

The average segmentation DSC for two readers in pairwise comparison was 0.919 for WG and 0.876 for TZ. Variability decreased with the number of readers: the interquartile ranges of the DSC were 0.076 (WG) / 0.021 (TZ) for configurations with two readers, 0.005 (WG) / 0.012 (TZ) for configurations with three readers, and 0.002 (WG) / 0.0037 (TZ) for configurations with six readers. The interquartile range decreased slightly faster between two and three readers than between three and six readers. When using consensus methods, variability often reached its minimum with three readers (with STAPLE, DSC = 0.96 [range: 0.945-0.971] for WG and DSC = 0.94 [range: 0.912-0.957] for TZ, and interquartile range was minimal for configurations with three readers.

CONCLUSION

The number of readers affects the inter-reader variability, in terms of inter-reader consistency and conformity to a reference. Variability is minimal for three readers, or three readers represent a tipping point in the variability evolution, with both pairwise-based metrics or metrics with respect to a reference. Accordingly, three readers may represent an optimal number to determine references for artificial intelligence applications.

Can We Identify Predictors of Success in Contouring Education for Radiation Oncology Trainees? An Analysis of the Anatomy and Radiology Contouring Bootcamp: Predictors of Success in Contouring Education

INTRODUCTION

Although several different contouring instructional programs are available to radiation oncologists and trainees, very little is known about which methods and resources benefit learners most, and whether some learners may need alternate forms of instruction. This study aimed to determine the factors that were predictors of learners' success in anatomy, radiology, and contouring education.

METHODS

Participants in the online and face-to-face (F2F) Anatomy and Radiology Contouring (ARC) Bootcamp completed pre- and post-intervention evaluations that assessed anatomy/radiology knowledge, contouring skills, self-confidence, and spatial ability. Baseline factors were assessed as predictors of outcomes across multiple educational domains.

RESULTS

One hundred and eighty (F2F: n=40; online: n=140) participants enrolled in the ARC Bootcamp and fifty-seven (F2F: n=30; online: n=27) participants completed both evaluations. Of the participants enrolled, 37% were female, and most were radiation oncology (RO) residents (62%). In the anatomy/radiology knowledge testing, all quartiles (based on baseline performance) improved numerically, however, the largest improvements occurred in learners with the lowest baseline scores (p<0.001). At the end of the Bootcamp, learners with lower-performing scores did not reach the level of learners with the highest baseline scores (Bonferroni-corrected p<0.001). Regarding the contouring assessment, improvements were only evident for the participants with lower-performing baseline scores (p<0.05). Spatial anatomy skills, as measured by the spatial anatomy task, was correlated to contouring ability. Overall, the greatest improvements were seen for learners in postgraduate year 1-3, those with no previous rotation experience in a given discipline, and those who attended from 'other' programs (i.e. medical physics residents and medical students).

CONCLUSIONS

The ARC Bootcamp improved all levels of performers' anatomy and radiology knowledge but only lower-performers' contouring ability. The course alone does not help lower-performing learners reach the abilities of higher-performers. The ARC Bootcamp tends to be most beneficial for participants with less RO experience. Curriculum modifications can be made to help support ARC Bootcamp participants with lower performing scores.

Evaluating the clinical acceptability of deep learning contours of prostate and organs-at-risk in an automated prostate treatment planning process

BACKGROUND

Radiation treatment is considered an effective and the most common treatment option for prostate cancer. The treatment planning process requires accurate and precise segmentation of the prostate and organs at risk (OARs), which is laborious and time-consuming when contoured manually. Artificial intelligence (AI)-based auto-segmentation has the potential to significantly accelerate the radiation therapy treatment planning process; however, the accuracy of auto-segmentation needs to be validated before its full clinical adoption.

PURPOSE

A commercial AI-based contouring model was trained to provide segmentation of the prostate and surrounding OARs. The segmented structures were input to a commercial auto-planning module for automated prostate treatment planning. This study comprehensively evaluates the performance of this contouring model in the automated prostate treatment planning process.

METHODS AND MATERIALS

A 3D U-Net-based model (INTContour, Carina AI) was trained and validated on 84 computed tomography (CT) scans and tested on an additional 23 CT scans from patients treated in our local institution. Prostate and OARs contours generated by the AI model (AI contour) were geometrically evaluated against Reference contours. The prostate contours were further evaluated against AI, Reference, and two additional observer contours for comparison using inter-observer variation (IOV) and 3D boundaries discrepancy analyses. A blinded evaluation was introduced to assess subjectively the clinical acceptability of the AI contours. Finally, treatment plans were created from an automated prostate planning workflow using the AI contours and were evaluated for their clinical acceptability following the RTOG-0815 protocol.

RESULTS

The AI contours demonstrated good geometric accuracy on OARs and prostate contours, with average Dice similarity coefficients (DSC) for bladder, rectum, femoral heads, seminal vesicles, and penile bulb of 0.93, 0.85, 0.96, 0.72, and 0.53, respectively. The DSC, 95% directed Hausdorff Distance (HD95), and Mean Surface Distance (MSD) for the prostate were 0.83±0.05, 6.07±1.87 mm, and 2.07±0.73 mm, respectively. No significant differences were found when comparing with IOV. In the double-blinded evaluation, 95.7% of the AI contours were scored as either "Perfect" (34.8%) or "Acceptable" (60.9%), while only one case (4.3%) was scored as "Unacceptable with minor changes required". In total, 69.6% of the AI contours were considered equal to or better than the Reference contours by an independent radiation oncologist. Automated treatment plans created from the AI contours produced similar and clinically-acceptable dosimetric distributions as those from plans created from Reference contours.

CONCLUSIONS

The investigated AI-based commercial model for prostate segmentation demonstrated good performance in clinical practice. Using this model, the implementation of an automated prostate treatment planning process is clinically feasible. This article is protected by copyright. All rights reserved.

A novel semi auto-segmentation method for accurate dose and NTCP evaluation in adaptive head and neck radiotherapy

BACKGROUND AND PURPOSE

Accurate segmentation of organs-at-risk (OARs) is crucial but tedious and time-consuming in adaptive radiotherapy (ART). The purpose of this work was to automate head and neck OAR-segmentation on repeat CT (rCT) by an optimal combination of human and auto-segmentation for accurate prediction of Normal Tissue Complication Probability (NTCP).

MATERIALS AND METHODS

Human segmentation (HS) of 3 observers, deformable image registration (DIR) based contour propagation and deep learning contouring (DLC) were carried out to segment 15 OARs on 15 rCTs. The original treatment plan was re-calculated on rCT to obtain mean dose (D_mean) and consequent NTCP-predictions. The average D_mean and NTCP-predictions of the three observers were referred to as the gold standard to calculate the absolute difference of D_mean and NTCP-predictions (|ΔD_mean| and |ΔNTCP|).

RESULTS

The average |ΔD_mean| of parotid glands in HS was 1.40 Gy, lower than that obtained with DIR and DLC (3.64 Gy, p < 0.001 and 3.72 Gy, p < 0.001, respectively). DLC showed the highest |ΔD_mean| in middle Pharyngeal Constrictor Muscle (PCM) (5.13 Gy, p = 0.01). DIR showed second highest |ΔD_mean| in the cricopharyngeal inlet (2.85 Gy, p = 0.01). The semi auto-segmentation (SAS) adopted HS, DIR and DLC for segmentation of parotid glands, PCM and all other OARs, respectively. The 90th percentile |ΔNTCP|was 2.19%, 2.24%, 1.10% and 1.50% for DIR, DLC, HS and SAS respectively.

CONCLUSIONS

Human segmentation of the parotid glands remains necessary for accurate interpretation of mean dose and NTCP during ART. Proposed semi auto-segmentation allows NTCP-predictions within 1.5% accuracy for 90% of the cases.

Implementation and evaluation of an online anatomy, radiology and contouring bootcamp for radiation therapists

BACKGROUND

As new treatments and technologies have been introduced in radiation oncology, the clinical roles of radiation therapists (RTs) have expanded. However, there are few formal learning opportunities for RTs. An online, anatomy, radiology and contouring bootcamp (ARC Bootcamp) originally designed for medical residents was identified as a prospective educational tool for RTs. The purpose of this study was to evaluate an RT edition of the ARC Bootcamp on knowledge, contouring, and confidence, as well as to identify areas for future modification.

METHODS

Fifty licensed RTs were enrolled in an eight-week, multidisciplinary, online RT ARC Bootcamp. Contouring practice was available throughout the course using an online contouring platform. Outcomes were evaluated using a pre-course and post-course multiple-choice quiz (MCQ), contouring evaluation and qualitative self-efficacy and satisfaction survey.

RESULTS

Of the fifty enrolled RTs, 30 completed the course, and 26 completed at least one of the post-tests. Nineteen contouring dice similarity coefficient (DSC) scores were available for paired pre- and post-course analysis. RTs demonstrated a statistically significant increase in mean DSC scoring pooled across all contouring structures (mean ± SD improvement: 0.09 ± 0.18 on a scale from 0 to 1, p=0.020). For individual contouring structures, 3/15 reached significance in contouring improvement. MCQ scores were available for 26 participants and increased after RT ARC Bootcamp participation with a mean ± SD pre-test score of 18.6 ± 4.2 (46.5%); on a 40-point scale vs. post-test score of 24.5 ± 4.3 (61.4%) (p < 0.001). RT confidence in contouring, anatomy knowledge and radiographic identification improved after course completion (p < 0.001). Feedback from RTs recommended more contouring instruction, less in-depth anatomy review and more time to complete the course.

CONCLUSIONS

The RT ARC Bootcamp was an effective tool for improving anatomy and radiographic knowledge among RTs. The course demonstrated improvements in contouring and overall confidence. However, only approximately half of the enrolled RTs completed the course, limiting statistical power. Future modifications will aim to increase relevance to RTs and improve completion rates.

Investigating the potential of deep learning for patient-specific quality assurance of salivary gland contours using EORTC-1219-DAHANCA-29 clinical trial data

INTRODUCTION

Manual quality assurance (QA) of radiotherapy contours for clinical trials is time and labor intensive and subject to inter-observer variability. Therefore, we investigated whether deep-learning (DL) can provide an automated solution to salivary gland contour QA.

MATERIAL AND METHODS

DL-models were trained to generate contours for parotid (PG) and submandibular glands (SMG). Sørensen-Dice coefficient (SDC) and Hausdorff distance (HD) were used to assess agreement between DL and clinical contours and thresholds were defined to highlight cases as potentially sub-optimal. 3 types of deliberate errors (expansion, contraction and displacement) were gradually applied to a test set, to confirm that SDC and HD were suitable QA metrics. DL-based QA was performed on 62 patients from the EORTC-1219-DAHANCA-29 trial. All highlighted contours were visually inspected.

RESULTS

Increasing the magnitude of all 3 types of errors resulted in progressively severe deterioration/increase in average SDC/HD. 19/124 clinical PG contours were highlighted as potentially sub-optimal, of which 5 (26%) were actually deemed clinically sub-optimal. 2/19 non-highlighted contours were false negatives (11%). 15/69 clinical SMG contours were highlighted, with 7 (47%) deemed clinically sub-optimal and 2/15 non-highlighted contours were false negatives (13%). For most incorrectly highlighted contours causes for low agreement could be identified.

CONCLUSION

Automated DL-based contour QA is feasible but some visual inspection remains essential. The substantial number of false positives were caused by sub-optimal performance of the DL-model. Improvements to the model will increase the extent of automation and reliability, facilitating the adoption of DL-based contour QA in clinical trials and routine practice.

Influence of inter-observer delineation variability on radiomic features of the parotid gland

PURPOSE

This study aimed to quantify the variability in the values of radiomic features extracted from a right parotid gland (RPG) delineated by a series of independent observers.

METHODS

This was a secondary analysis of anonymous data from a delineation workshop. Inter-observer variability of the RPG from 40 participants was quantified using DICE similarity coefficient (DSC) and Hausdorff distance (HD). An additional contour was generated using Varian SmartSegmentation. Radiomic features extracted include four shape features, six histogram features, and 32 texture features. The absolute mean paired percentage difference (PPD) in feature values from the expert and participants were ranked . Feature robustness was classified using pre- determined thresholds.

RESULTS

63% of participants achieved a DSC > 0.7, the auto- segmentation DSC was 0.76. The average HD for the participants was 16.16 mm ± 0.66 mm, and 15.16 mm for the auto-segmentation. 48% (n = 20) and 33% (n = 14) of features were deemed to be robust with a mean absolute PPD < 5%, for the auto-segmentation and manual delineations respectively; the majority of which were from the grey-run length matrix family. 7% (n = 3) of features from the auto- segmentation and 10% (n = 4) from the manual contours were deemed to be unstable with a mean absolute PPD > 50%. The value of the most robust feature was not related to DSC and HD.

CONCLUSION

Inter-observer delineation variability affects the value of the radiomic features extracted from the RPG. This study identifies the radiomic features least sensitive to these uncertainties. Further investigation of the clinical relevance of these features in prediction of xerostomia is warranted.

Auto-segmentation of organs at risk for head and neck radiotherapy planning: From atlas-based to deep learning methods

Radiotherapy (RT) is one of the basic treatment modalities for cancer of the head and neck (H&N), which requires a precise spatial description of the target volumes and organs at risk (OARs) to deliver a highly conformal radiation dose to the tumor cells while sparing the healthy tissues. For this purpose, target volumes and OARs have to be delineated and segmented from medical images. As manual delineation is a tedious and time-consuming task subjected to intra/interobserver variability, computerized auto-segmentation has been developed as an alternative. The field of medical imaging and RT planning has experienced an increased interest in the past decade, with new emerging trends that shifted the field of H&N OAR auto-segmentation from atlas-based to deep learning-based approaches. In this review, we systematically analyzed 78 relevant publications on auto-segmentation of OARs in the H&N region from 2008 to date, and provided critical discussions and recommendations from various perspectives: image modality - both computed tomography and magnetic resonance image modalities are being exploited, but the potential of the latter should be explored more in the future; OAR - the spinal cord, brainstem, and major salivary glands are the most studied OARs, but additional experiments should be conducted for several less studied soft tissue structures; image database - several image databases with the corresponding ground truth are currently available for methodology evaluation, but should be augmented with data from multiple observers and multiple institutions; methodology - current methods have shifted from atlas-based to deep learning auto-segmentation, which is expected to become even more sophisticated; ground truth - delineation guidelines should be followed and participation of multiple experts from multiple institutions is recommended; performance metrics - the Dice coefficient as the standard volumetric overlap metrics should be accompanied with at least one distance metrics, and combined with clinical acceptability scores and risk assessments; segmentation performance - the best performing methods achieve clinically acceptable auto-segmentation for several OARs, however, the dosimetric impact should be also studied to provide clinically relevant endpoints for RT planning.

A Systematic Review of Contouring Guidelines in Radiation Oncology: Analysis of Frequency, Methodology, and Delivery of Consensus Recommendations

PURPOSE

Clinical trials have described variation in radiation therapy plan quality, of which contour delineation is a key component, and linked this to inferior patient outcomes. In response, consensus guidelines have been developed to standardize contour delineation. This investigation assesses trends in contouring guidelines and examines the methodologies used to generate and deliver recommendations.

METHODS AND MATERIALS

We conducted a literature search for contouring guidelines published after 1995. Of 11,124 citations, 332 were identified for full-text review to determine inclusion. We abstracted articles for the intent of the consensus process, key elements of the methodology, and mode of information delivery. A Fisher exact test was used to identify elements that differed among the guidelines generated for clinical trials and routine care.

RESULTS

Overall, 142 guidelines were included, of which 16 (11%) were developed for a clinical trial. There was an increase in guideline publication over time (0 from 1995-1999 vs 65 from 2015- 2019; P = .03), particularly among recommendations for stereotactic radiation and brachytherapy. The most common disease sites were head and neck (24%), gastrointestinal (12%), and gynecologic (12%). Methods used to develop recommendations included literature review (50%) and image-based methods (45%). Panels included a median of 10 physicians (interquartile range, 7-16); 70% of panels represented multidisciplinary expertise. Guidelines developed for a clinical trial were more likely to include an image-based approach, with quantitative analysis of contours submitted by the panel members and to publish a full set of image-based recommendations (P < .005).

CONCLUSIONS

This review highlights an increase in consensus contouring recommendations over time. Guidelines focus on disease sites, such as head and neck, with evidence supporting a correlation between treatment planning and patient outcomes, although variation exists in the approach to the consensus process. Elements that may improve guideline acceptance (ie, image-based consensus contour analysis) and usability (ie, inclusion of a full image set) are more common in guidelines developed for clinical trials.

Automated Quality Assurance of OAR Contouring for Lung Cancer Based on Segmentation With Deep Active Learning

Purpose: Ensuring high-quality data for clinical trials in radiotherapy requires the generation of contours that comply with protocol definitions. The current workflow includes a manual review of the submitted contours, which is time-consuming and subjective. In this study, we developed an automated quality assurance (QA) system for lung cancer based on a segmentation model trained with deep active learning.

Methods: The data included a gold atlas with 36 cases and 110 cases from the “NRG Oncology/RTOG 1308 Trial”. The first 70 cases enrolled to the RTOG 1308 formed the candidate set, and the remaining 40 cases were randomly assigned to validation and test sets (each with 20 cases). The organs-at-risk included the heart, esophagus, spinal cord, and lungs. A preliminary convolutional neural network segmentation model was trained with the gold standard atlas. To address the deficiency of the limited training data, we selected quality images from the candidate set to be added to the training set for fine-tuning of the model with deep active learning. The trained robust segmentation models were used for QA purposes. The segmentation evaluation metrics derived from the validation set, including the Dice and Hausdorff distance, were used to develop the criteria for QA decision making. The performance of the strategy was assessed using the test set.

Results: The QA method achieved promising contouring error detection, with the following metrics for the heart, esophagus, spinal cord, left lung, and right lung: balanced accuracy, 0.96, 0.95, 0.96, 0.97, and 0.97, respectively; sensitivity, 0.95, 0.98, 0.96, 1.0, and 1.0, respectively; specificity, 0.98, 0.92, 0.97, 0.94, and 0.94, respectively; and area under the receiving operator characteristic curve, 0.96, 0.95, 0.96, 0.97, and 0.94, respectively.

Conclusions: The proposed system automatically detected contour errors for QA. It could provide consistent and objective evaluations with much reduced investigator intervention in multicenter clinical trials.

Systematic review of educational interventions to improve contouring in radiotherapy

BACKGROUND AND PURPOSE

Contouring is a critical step in the radiotherapy process, but there is limited research on how to teach it and no consensus about the best method. We summarize the current evidence regarding improvement of contouring skills.

METHODS AND MATERIALS

Comprehensive literature search of the Pubmed-MEDLINE database, EMBASE database and Cochrane Library to identify relevant studies (independently examined by two investigators) that included baseline contouring followed by a re-contouring assessment after an educational intervention.

RESULTS

598 papers were identified. 16 studies met the inclusion criteria representing 370 participants (average number of participants per study of 23; range (4-141). Regarding the teaching methodology, 5/16 used onsite courses, 8/16 online courses, and 2/16 used blended learning. Study quality was heterogenous. There were only 3 randomized studies and only 3 analyzed the dosimetric impact of improving contouring homogeneity. Dice similarity coefficient was the most common evaluation metric (7/16), and in all these studies at least some contours improved significantly post-intervention. The time frame for evaluating the learning effect of the teaching intervention was almost exclusively short-time, with only one study evaluating the long-term utility of the educational program beyond 6 months.

CONCLUSION

The literature on educational interventions designed to improve contouring performance is limited and heterogenous. Onsite, online and blended learning courses have all been shown to be helpful, however, sample sizes are small and impact assessment is almost exclusively short-term and typically does not take into account the effect on treatment planning. The most effective teaching methodology/format is unknown and impact on daily clinical practice is uncertain.

[Target volume concepts in radiotherapy and their implications for imaging]

CLINICAL ISSUE

Successful radiotherapy requires precise localization of the tumor and requires high-quality imaging for developing a treatment plan.

STANDARD TREATMENT

Irradiation of the tumor region, including a safety margin.

TREATMENT INNOVATIONS

The target volume consists of the gross tumor volume (GTV) containing visible parts of the tumor, the clinical target volume (CTV) covering the GTV plus invisible tumor extensions, and the planning target volume (PTV) to account for uncertainties. The non-GTV parts of the CTV are based on historical patient data. The PTV margins are based on a calculation of possible uncertainties during planning, setup, or treatment. Normal tissue deserves the identical care in contouring, since its tolerance may limit the tumor dose, taking into account the contours of organs at risk. Serial risk organs benefit from defining a planning organ of risk volume (PRV) to better limit the dose delivered to them.

DIAGNOSTIC WORK-UP

The better the imaging, the more reliable the definition of the GTV and treatment success will be. Multiple imaging sequences are desirable to support the delineation of the tumor. They may result in different CTVs that, depending on their tumor burden, may require different doses.

PERFORMANCE

The definition of standardized target volumes according to the ICRU reports 50, 62, and 83 forms the basis for an individualized radiation treatment planning according to unified criteria on a high-quality level.

ACHIEVEMENTS

Radio-oncology is by nature interdisciplinary, the diagnostic radiologist being an indispensable team partner. A regular dialogue between the disciplines is pivotal for target volume definition and treatment success.

PRACTICAL RECOMMENDATIONS

Imaging for target volume definition requires highest quality imaging, the use of functional imaging methods and close cooperation with a diagnostic radiologist experienced in this field.

Quantifying the dosimetric impact of organ-at-risk delineation variability in head and neck radiation therapy in the context of patient setup uncertainty

The purpose of this study was to quantify the potential dosimetric impact of delineation variability (DV) in head and neck radiation therapy (RT) when inherent patient setup variability (SV) is also considered. The impact of DV was assessed by generating plans with multiple structure sets, cross-evaluating them, including SV, across sets, and determining P _PQM: the probability of achieving organ-specific plan quality metrics (PQM). DV was incorporated by: (1) using multiple organ at risk (OAR) structure sets delineated by independent manual observers; and (2) randomly perturbing manually generated OARs to generate alternatives with varying levels of uncertainty (low, medium, and high DV). For each structure set, independent VMAT plans were auto-generated to meet clinical PQMs. Each plan was cross-evaluated using OARs from multiple structure sets with simulated SV including per-fraction random (σ _s) and per-treatment-course systematic (Σ_s) setup errors. The dosimetric impact of DV was assessed by examining P _PQM with and without SV/DV. Clinically significant differences were defined by those that exceeded differences caused by a  +2% output variation. Without including SV, simulated DV at the medium level reduced P _PQM by an average of 5.5% for all OARs with D _max PQMs. This reduction decreased to 2.8% for SV  =  2 mm and 2.4% for SV  =  4 mm (the average P _PQM reduction due to 2% output errors was 2.7%). For OARs with D _mean PQMs, the average P _PQM reduction was 0.9% for SV  =  0 and  ⩽0.1% for SV  ⩾  2 mm. The effect of DV was larger for OARs that directly abutted a target volume than for those that did not. These trends were also observed with real DV from multi-observer delineations. The dosimetric impact of DV appeared to decrease when random and systematic SV was considered. Sensitivity to DV was affected by OAR objective type (i.e. D _mean versus D _max objectives) as well as distance from the target volume.

Inter-observer variability of clinical target volume delineation in definitive radiotherapy of neck lymph node metastases from unknown primary. A cooperative study of the Italian Association of Radiotherapy and Clinical Oncology (AIRO) Head and Neck Group

BACKGROUND

This study, promoted by Italian Association of Radiotherapy and Clinical Oncology (AIRO) Head and Neck Group, aimed to assess the current national practice of target volume delineation on a case of neck lymph node metastases from unknown primary evaluating inter-observer variability, in a setting of primary radiotherapy.

MATERIALS AND METHODS

A case of metastatic neck lymph node from occult primary was proposed to 17 radiation oncologists. A national reference RT center was identified and considered as benchmark. Participants were requested to delineate target volumes. A structured questionnaire was administered. A comparison between following parameters of the CTVs was performed: centroids distances, Dice similarity index (DSI), Jaccard index and mean distance to agreement (MDA). Volume expressed in cubic centimeters and CTVs cranio-caudal extension were evaluated.

RESULTS

Sixteen of 17 radiation oncologists recommended three CTVs dose levels. (CTV HD, CTV ID and CTV LD); CTV ID was not delineated by one of the participants and by the reference center. The distance between the reference centroid and the mean centroid of CTVs HD was 1.09 cm (0.36-3.99 cm); for CTV LD, a mean centroids distance of 2.45 (0.27-4.83 cm) was found, and for CTV HD, mean DSI is 0.48 and mean Jaccard index is 0.32 and MDA was 8.89 mm. CTV LD showed a mean DSI of 0.46, mean Jaccard index of 0.31 and MDA of 14.87 when compared to the reference.

CONCLUSION

Many aspects concerning treatment optimization of cervical nodes metastases from occult primary remain unclear, and we found a notable heterogeneity of global radiotherapy management reporting discordances both in target volume delineation and volume prescription.

Use of a novel atlas for muscles of mastication to reduce inter observer variability in head and neck radiotherapy contouring

PURPOSE/OBJECTIVE(S)

Trismus is caused by injury to the masticatory muscles resulting from cancer or its treatment. Contouring these muscles to reduce dose and radiation related trismus can be problematic due to interobserver variability. This study aimed to evaluate the reduction in interobserver variability achievable with a new contouring atlas.

MATERIALS/METHODS

The atlas included: medial and lateral pterygoids (MP, LP), masseter (M) and temporalis (T) muscles, and the temporo-mandibular joint (TMJ). Seven clinicians delineated five paired structures on CT scans from 5 patients without the atlas. After ≥5 weeks, contouring was repeated using the atlas. Using contours generated by the clinicians on the same 5 CT scans as reference, dice similarity coefficient (DSC), mean distance-to-agreement (DTA) and centre of mass (COM) difference were compared with and without the atlas. Comparison was also performed split by training grade. Mean and standard deviation (SD) values were measured.

RESULTS

The atlas reduced interobserver variability for all structures. Mean DTA significantly improved for MP (p = 0.01), M (p < 0.01), T (p < 0.01) and TMJ (p < 0.01). Mean DTA improved using the atlas for the trainees across all muscles, with the largest reduction in variability observed for the T (4.3 ± 7.1 v 1.2 ± 0.4 mm, p = 0.06) and TMJ (2.1 ± 0.7 v 0.8 ± 0.3 mm, p < 0.01). Distance between the COM and interobserver variability reduced in all directions for MP and T.

CONCLUSION

A new atlas for contouring masticatory muscles during radiotherapy planning for head and neck cancer reduces interobserver variability and could be used as an educational tool.

Evaluation of diffusion weighted imaging for tumor delineation in head-and-neck radiotherapy by comparison with automatically segmented 18F-fluorodeoxyglucose positron emission tomography

Background and purpose

Diffusion weighted (DW) MRI may facilitate target volume delineation for head-and-neck (HN) radiation treatment planning. In this study we assessed the use of a dedicated, geometrically accurate, DW-MRI sequence for target volume delineation. The delineations were compared with semi-automatic segmentations on ¹⁸F-fluorodeoxyglucose (FDG) positron emission tomography (PET) images and evaluated for interobserver variation.

Methods and materials

Fifteen HN cancer patients underwent both DW-MRI and FDG-PET for RT treatment planning. Target delineation on DW-MRI was performed by three observers, while for PET a semi-automatic segmentation was performed using a Gaussian mixture model. For interobserver variation and intermodality variation, volumes, overlap metrics and Hausdorff distances were calculated from the delineations.

Results

The median volumes delineated by the three observers on DW-MRI were 10.8, 10.5 and 9.0 cm³ respectively, and was larger than the median PET volume (8.0 cm³). The median conformity index of DW-MRI for interobserver variation was 0.73 (range 0.38–0.80). Compared to PET, the delineations on DW-MRI by the three observers showed a median dice similarity coefficient of 0.71, 0.69 and 0.72 respectively. The mean Hausdorff distance was small with median (range) distances between PET and DW-MRI of 2.3 (1.5–6.8), 2.5 (1.6–6.9) and 2.0 (1.35–7.6) mm respectively. Over all patients, the median 95th percentile distances were 6.0 (3.0–13.4), 6.6 (4.0–24.0) and 5.3 (3.4–26.0) mm.

Conclusion

Using a dedicated DW-MRI sequence, target volumes could be defined with good interobserver agreement and a good overlap with PET. Target volume delineation using DW-MRI is promising in head-and-neck radiotherapy, combined with other modalities, it can lead to more precise target volume delineation.

Observer variation in the delineation of organs at risk for head and neck radiation therapy treatment planning: a systematic review protocol

REVIEW QUESTION

The objective of this review is to examine inter- and intra-observer agreement and reliability in the delineation of head and neck organs at risk (OAR) as part of the radiation therapy treatment planning process.More specifically, the objectives are to identify.

Challenges for Quality Assurance of Target Volume Delineation in Clinical Trials

In recent years, new radiotherapy techniques have emerged that aim to improve treatment outcome and reduce toxicity. The standard method of evaluating such techniques is to conduct large scale multicenter clinical trials, often across continents. A major challenge for such trials is quality assurance to ensure consistency of treatment across all participating centers. Analyses from previous studies have shown that poor compliance and protocol violation have a significant adverse effect on treatment outcomes. The results of the clinical trials may, therefore, be confounded by poor quality radiotherapy. Target volume delineation (TVD) is one of the most critical steps in the radiotherapy process. Many studies have shown large inter-observer variations in contouring, both within and outside of clinical trials. High precision techniques, such as intensity-modulated radiotherapy, image-guided brachytherapy, and stereotactic radiotherapy have steep dose gradients, and errors in contouring may lead to inadequate dose to the tumor and consequently, reduce the chance of cure. Similarly, variation in organ at risk delineation will make it difficult to evaluate dose response for toxicity. This article reviews the literature on TVD variability and its impact on dosimetry and clinical outcomes. The implications for quality assurance in clinical trials are discussed.

A Comparative Evaluation of 3 Different Free-Form Deformable Image Registration and Contour Propagation Methods for Head and Neck MRI: The Case of Parotid Changes During Radiotherapy

Purpose:

To validate and compare the deformable image registration and parotid contour propagation process for head and neck magnetic resonance imaging in patients treated with radiotherapy using 3 different approaches—the commercial MIM, the open-source Elastix software, and an optimized version of it.

Materials and Methods:

Twelve patients with head and neck cancer previously treated with radiotherapy were considered. Deformable image registration and parotid contour propagation were evaluated by considering the magnetic resonance images acquired before and after the end of the treatment. Deformable image registration, based on free-form deformation method, and contour propagation available on MIM were compared to Elastix. Two different contour propagation approaches were implemented for Elastix software, a conventional one (DIR_Trx) and an optimized homemade version, based on mesh deformation (DIR_Mesh). The accuracy of these 3 approaches was estimated by comparing propagated to manual contours in terms of average symmetric distance, maximum symmetric distance, Dice similarity coefficient, sensitivity, and inclusiveness.

Results:

A good agreement was generally found between the manual contours and the propagated ones, without differences among the 3 methods; in few critical cases with complex deformations, DIR_Mesh proved to be more accurate, having the lowest values of average symmetric distance and maximum symmetric distance and the highest value of Dice similarity coefficient, although nonsignificant. The average propagation errors with respect to the reference contours are lower than the voxel diagonal (2 mm), and Dice similarity coefficient is around 0.8 for all 3 methods.

Conclusion:

The 3 free-form deformation approaches were not significantly different in terms of deformable image registration accuracy and can be safely adopted for the registration and parotid contour propagation during radiotherapy on magnetic resonance imaging. More optimized approaches (as DIR_Mesh) could be preferable for critical deformations.

Uncertainties in volume delineation in radiation oncology: A systematic review and recommendations for future studies

BACKGROUND AND PURPOSE

Volume delineation is a well-recognised potential source of error in radiotherapy. Whilst it is important to quantify the degree of interobserver variability (IOV) in volume delineation, the resulting impact on dosimetry and clinical outcomes is a more relevant endpoint. We performed a literature review of studies evaluating IOV in target volume and organ-at-risk (OAR) delineation in order to analyse these with respect to the metrics used, reporting of dosimetric consequences, and use of statistical tests.

METHODS AND MATERIALS

Medline and Pubmed databases were queried for relevant articles using keywords. We included studies published in English between 2000 and 2014 with more than two observers.

RESULTS

119 studies were identified covering all major tumour sites. CTV (n=47) and GTV (n=38) were most commonly contoured. Median number of participants and data sets were 7 (3-50) and 9 (1-132) respectively. There was considerable heterogeneity in the use of metrics and methods of analysis. Statistical analysis of results was reported in 68% (n=81) and dosimetric consequences in 21% (n=25) of studies.

CONCLUSION

There is a lack of consistency in conducting and reporting analyses from IOV studies. We suggest a framework to use for future studies evaluating IOV.

The dosimetric impact of target volume delineation variation for cervical cancer radiotherapy

BACKGROUND

Cervical cancer inter-observer delineation variation has been demonstrated. This article addresses its dosimetric impact.

METHODS

21 centres outlined two INTERLACE trial quality assurance test cases. A gold standard clinical target volume (GSCTV) was created from a consensus and STAPLE outline. RapidArc plans were created for all centres' planning target volumes (PTVs; PTV1+2). Gold standard PTVs (GSPTVs) were created for each plan by applying each centre's CTV-PTV margins to GSCTV. DVH parameters including D95% and Dmean for each PTV1+2 and GSPTV were compared, representing planned versus GSPTV delivered dose. PTV1+2 and GSPTV V95% was also calculated.

RESULTS

Reviewing all parameters, no plans achieved acceptable GSPTV coverage. GSPTV V95%⩾95% was not achieved for any plan. GSPTV V95%<90% in 15/21 (case 1) and 14/22 (case 2) and <80% in 2 plans from both cases. GSPTV V95% is on average 10-15% lower than planned and GSPTV D95% is 10-20% lower than planned. Most common GSCTV anatomical areas not receiving 95% dose were vagina, obturator and external iliac nodes and, in case 1, the superior nodal aspect.

CONCLUSION

Cervical cancer CTV delineation variation leads to significant reductions in dose delivered to GSPTV. This highlights the ongoing importance of standardising delineation in the IMRT era.

Influence of the type of imaging on the delineation process during the treatment planning

AIM

The aim of this study was to compare the intra- and interobserver contouring variability for structures with density of organ at risk in two types of tomography: kilovoltage computed tomography (KVCT) versus megavoltage computed tomography (MVCT). The intra- and interobserver differences were examined on both types of tomography for structures which simulate human tissue or organs.

MATERIALS AND METHODS

Six structures with density of the liver, bone, trachea, lung, soft tissue and muscle were created and used. For the measurements, the special water phantom with all structures was designed. To evaluate interobserver variability, five observers delineated the structures in both types of computed tomography (CT).

RESULTS

Intraobserver variability was in the range of 1-14% and was the largest for the liver. The observers segmented larger volumes on MVCT compared with KVCT for the trachea (79.56 ccm vs.74.91 ccm), lung (87.61 vs. 82.50), soft tissue (154.24 vs. 145.47) and muscle (164.01 vs. 157.89). For the liver (98.13 vs. 99.38) and bone (51.86 vs. 67.97), the volume on MVCT was smaller than KVCT. The statistically significant differences between observers were observed for structures with density of the liver, bone and soft tissue on KVCT and for the liver, lung and soft tissue on MVCT. For the structures with density of the trachea and muscles, there were no significant differences for both types of tomography.

CONCLUSIONS

During the contouring process the interobserver and intraobserver contouring uncertainty was larger on MVCT, especially for structures with HU near 80, compared with KVCT.

Variation of normal tissue complication probability (NTCP) estimates of radiation-induced hypothyroidism in relation to changes in delineation of the thyroid gland

BACKGROUND

To examine the variations of risk-estimates of radiation-induced hypothyroidism (HT) from our previously developed normal tissue complication probability (NTCP) model in patients with head and neck squamous cell carcinoma (HNSCC) in relation to variability of delineation of the thyroid gland.

PATIENTS AND METHODS

In a previous study for development of an NTCP model for HT, the thyroid gland was delineated in 246 treatment plans of patients with HNSCC. Fifty of these plans were randomly chosen for re-delineation for a study of the intra- and inter-observer variability of thyroid volume, Dmean and estimated risk of HT. Bland-Altman plots were used for assessment of the systematic (mean) and random [standard deviation (SD)] variability of the three parameters, and a method for displaying the spatial variation in delineation differences was developed.

RESULTS

Intra-observer variability resulted in a mean difference in thyroid volume and Dmean of 0.4 cm(3) (SD ± 1.6) and -0.5 Gy (SD ± 1.0), respectively, and 0.3 cm(3) (SD ± 1.8) and 0.0 Gy (SD ± 1.3) for inter-observer variability. The corresponding mean differences of NTCP values for radiation-induced HT due to intra- and inter-observer variations were insignificantly small, -0.4% (SD ± 6.0) and -0.7% (SD ± 4.8), respectively, but as the SDs show, for some patients the difference in estimated NTCP was large.

CONCLUSION

For the entire study population, the variation in predicted risk of radiation-induced HT in head and neck cancer was small and our NTCP model was robust against observer variations in delineation of the thyroid gland. However, for the individual patient, there may be large differences in estimated risk which calls for precise delineation of the thyroid gland to obtain correct dose and NTCP estimates for optimized treatment planning in the individual patient.

Vision 20/20: perspectives on automated image segmentation for radiotherapy

Due to rapid advances in radiation therapy (RT), especially image guidance and treatment adaptation, a fast and accurate segmentation of medical images is a very important part of the treatment. Manual delineation of target volumes and organs at risk is still the standard routine for most clinics, even though it is time consuming and prone to intra- and interobserver variations. Automated segmentation methods seek to reduce delineation workload and unify the organ boundary definition. In this paper, the authors review the current autosegmentation methods particularly relevant for applications in RT. The authors outline the methods' strengths and limitations and propose strategies that could lead to wider acceptance of autosegmentation in routine clinical practice. The authors conclude that currently, autosegmentation technology in RT planning is an efficient tool for the clinicians to provide them with a good starting point for review and adjustment. Modern hardware platforms including GPUs allow most of the autosegmentation tasks to be done in a range of a few minutes. In the nearest future, improvements in CT-based autosegmentation tools will be achieved through standardization of imaging and contouring protocols. In the longer term, the authors expect a wider use of multimodality approaches and better understanding of correlation of imaging with biology and pathology.

Parotid glands dose-effect relationships based on their actually delivered doses: implications for adaptive replanning in radiation therapy of head-and-neck cancer

PURPOSE

Doses actually delivered to the parotid glands during radiation therapy often exceed planned doses. We hypothesized that the delivered doses correlate better with parotid salivary output than the planned doses, used in all previous studies, and that determining these correlations will help make decisions regarding adaptive radiation therapy (ART) aimed at reducing the delivered doses.

METHODS AND MATERIALS

In this prospective study, oropharyngeal cancer patients treated definitively with chemoirradiation underwent daily cone-beam computed tomography (CBCT) with clinical setup alignment based on the C2 posterior edge. Parotid glands in the CBCTs were aligned by deformable registration to calculate cumulative delivered doses. Stimulated salivary flow rates were measured separately from each parotid gland pretherapy and periodically posttherapy.

RESULTS

Thirty-six parotid glands of 18 patients were analyzed. Average mean planned doses was 32 Gy, and differences from planned to delivered mean gland doses were -4.9 to +8.4 Gy, median difference +2.2 Gy in glands in which delivered doses increased relative to planned. Both planned and delivered mean doses were significantly correlated with posttreatment salivary outputs at almost all posttherapy time points, without statistically significant differences in the correlations. Large dispersions (on average, SD 3.6 Gy) characterized the dose-effect relationships for both. The differences between the cumulative delivered doses and planned doses were evident at first fraction (r=.92, P<.0001) because of complex setup deviations (eg, rotations and neck articulations), uncorrected by the translational clinical alignments.

CONCLUSIONS

After daily translational setup corrections, differences between planned and delivered doses in most glands were small relative to the SDs of the dose-saliva data, suggesting that ART is not likely to gain measurable salivary output improvement in most cases. These differences were observed at first treatment, indicating potential benefit for more complex setup corrections or adaptive interventions in the minority of patients with large deviations detected early by CBCT.