A gel tumour phantom for assessment of the accuracy of manual and automatic delineation of gross tumour volume from FDG-PET/CT

Classification and evaluation strategies of auto-segmentation approaches for PET: Report of AAPM task group No. 211

PURPOSE

The purpose of this educational report is to provide an overview of the present state-of-the-art PET auto-segmentation (PET-AS) algorithms and their respective validation, with an emphasis on providing the user with help in understanding the challenges and pitfalls associated with selecting and implementing a PET-AS algorithm for a particular application.

APPROACH

A brief description of the different types of PET-AS algorithms is provided using a classification based on method complexity and type. The advantages and the limitations of the current PET-AS algorithms are highlighted based on current publications and existing comparison studies. A review of the available image datasets and contour evaluation metrics in terms of their applicability for establishing a standardized evaluation of PET-AS algorithms is provided. The performance requirements for the algorithms and their dependence on the application, the radiotracer used and the evaluation criteria are described and discussed. Finally, a procedure for algorithm acceptance and implementation, as well as the complementary role of manual and auto-segmentation are addressed.

FINDINGS

A large number of PET-AS algorithms have been developed within the last 20 years. Many of the proposed algorithms are based on either fixed or adaptively selected thresholds. More recently, numerous papers have proposed the use of more advanced image analysis paradigms to perform semi-automated delineation of the PET images. However, the level of algorithm validation is variable and for most published algorithms is either insufficient or inconsistent which prevents recommending a single algorithm. This is compounded by the fact that realistic image configurations with low signal-to-noise ratios (SNR) and heterogeneous tracer distributions have rarely been used. Large variations in the evaluation methods used in the literature point to the need for a standardized evaluation protocol.

CONCLUSIONS

Available comparison studies suggest that PET-AS algorithms relying on advanced image analysis paradigms provide generally more accurate segmentation than approaches based on PET activity thresholds, particularly for realistic configurations. However, this may not be the case for simple shape lesions in situations with a narrower range of parameters, where simpler methods may also perform well. Recent algorithms which employ some type of consensus or automatic selection between several PET-AS methods have potential to overcome the limitations of the individual methods when appropriately trained. In either case, accuracy evaluation is required for each different PET scanner and scanning and image reconstruction protocol. For the simpler, less robust approaches, adaptation to scanning conditions, tumor type, and tumor location by optimization of parameters is necessary. The results from the method evaluation stage can be used to estimate the contouring uncertainty. All PET-AS contours should be critically verified by a physician. A standard test, i.e., a benchmark dedicated to evaluating both existing and future PET-AS algorithms needs to be designed, to aid clinicians in evaluating and selecting PET-AS algorithms and to establish performance limits for their acceptance for clinical use. The initial steps toward designing and building such a standard are undertaken by the task group members.

Multi-site quality and variability analysis of 3D FDG PET segmentations based on phantom and clinical image data

PURPOSE

Radiomics utilizes a large number of image-derived features for quantifying tumor characteristics that can in turn be correlated with response and prognosis. Unfortunately, extraction and analysis of such image-based features is subject to measurement variability and bias. The challenge for radiomics is particularly acute in Positron Emission Tomography (PET) where limited resolution, a high noise component related to the limited stochastic nature of the raw data, and the wide variety of reconstruction options confound quantitative feature metrics. Extracted feature quality is also affected by tumor segmentation methods used to define regions over which to calculate features, making it challenging to produce consistent radiomics analysis results across multiple institutions that use different segmentation algorithms in their PET image analysis. Understanding each element contributing to these inconsistencies in quantitative image feature and metric generation is paramount for ultimate utilization of these methods in multi-institutional trials and clinical oncology decision making.

METHODS

To assess segmentation quality and consistency at the multi-institutional level, we conducted a study of seven institutional members of the National Cancer Institute Quantitative Imaging Network. For the study, members were asked to segment a common set of phantom PET scans acquired over a range of imaging conditions as well as a second set of head and neck cancer (HNC) PET scans. Segmentations were generated at each institution using their preferred approach. In addition, participants were asked to repeat segmentations with a time interval between initial and repeat segmentation. This procedure resulted in overall 806 phantom insert and 641 lesion segmentations. Subsequently, the volume was computed from the segmentations and compared to the corresponding reference volume by means of statistical analysis.

RESULTS

On the two test sets (phantom and HNC PET scans), the performance of the seven segmentation approaches was as follows. On the phantom test set, the mean relative volume errors ranged from 29.9 to 87.8% of the ground truth reference volumes, and the repeat difference for each institution ranged between -36.4 to 39.9%. On the HNC test set, the mean relative volume error ranged between -50.5 to 701.5%, and the repeat difference for each institution ranged between -37.7 to 31.5%. In addition, performance measures per phantom insert/lesion size categories are given in the paper. On phantom data, regression analysis resulted in coefficient of variation (CV) components of 42.5% for scanners, 26.8% for institutional approaches, 21.1% for repeated segmentations, 14.3% for relative contrasts, 5.3% for count statistics (acquisition times), and 0.0% for repeated scans. Analysis showed that the CV components for approaches and repeated segmentations were significantly larger on the HNC test set with increases by 112.7% and 102.4%, respectively.

CONCLUSION

Analysis results underline the importance of PET scanner reconstruction harmonization and imaging protocol standardization for quantification of lesion volumes. In addition, to enable a distributed multi-site analysis of FDG PET images, harmonization of analysis approaches and operator training in combination with highly automated segmentation methods seems to be advisable. Future work will focus on quantifying the impact of segmentation variation on radiomics system performance.

Diffusion-weighted magnetic resonance imaging during radiotherapy of locally advanced cervical cancer--treatment response assessment using different segmentation methods

BACKGROUND

Diffusion-weighted magnetic resonance imaging (DW-MRI) and the derived apparent diffusion coefficient (ADC) value has potential for monitoring tumor response to radiotherapy (RT). Method used for segmentation of volumes with reduced diffusion will influence both volume size and observed distribution of ADC values. This study evaluates: 1) different segmentation methods; and 2) how they affect assessment of tumor ADC value during RT.

MATERIAL AND METHODS

Eleven patients with locally advanced cervical cancer underwent MRI three times during their RT: prior to start of RT (PRERT), two weeks into external beam RT (WK2RT) and one week prior to brachytherapy (PREBT). Volumes on DW-MRI were segmented using three semi-automatic segmentation methods: "cluster analysis", "relative signal intensity (SD4)" and "region growing". Segmented volumes were compared to the gross tumor volume (GTV) identified on T2-weighted MR images using the Jaccard similarity index (JSI). ADC values from segmented volumes were compared and changes of ADC values during therapy were evaluated.

RESULTS

Significant difference between the four volumes (GTV, DWIcluster, DWISD4 and DWIregion) was found (p < 0.01), and the volumes changed significantly during treatment (p < 0.01). There was a significant difference in JSI among segmentation methods at time of PRERT (p < 0.016) with region growing having the lowest JSIGTV (mean± sd: 0.35 ± 0.1), followed by the SD4 method (mean± sd: 0.50 ± 0.1) and clustering (mean± sd: 0.52 ± 0.3). There was no significant difference in mean ADC value compared at same treatment time. Mean tumor ADC value increased significantly (p < 0.01) for all methods across treatment time.

CONCLUSION

Among the three semi-automatic segmentations of hyper-intense intensities on DW-MR images implemented, cluster analysis and relative signal thresholding had the greatest similarity to the clinical tumor volume. Evaluation of mean ADC value did not depend on segmentation method.