Is STAPLE algorithm confident to assess segmentation methods in PET imaging?

Automated hippocampal segmentation algorithms evaluated in stroke patients

Deep learning segmentation algorithms can produce reproducible results in a matter of seconds. However, their application to more complex datasets is uncertain and may fail in the presence of severe structural abnormalities-such as those commonly seen in stroke patients. In this investigation, six recent, deep learning-based hippocampal segmentation algorithms were tested on 641 stroke patients of a multicentric, open-source dataset ATLAS 2.0. The comparisons of the volumes showed that the methods are not interchangeable with concordance correlation coefficients from 0.266 to 0.816. While the segmentation algorithms demonstrated an overall good performance (volumetric similarity [VS] 0.816 to 0.972, DICE score 0.786 to 0.921, and Hausdorff distance [HD] 2.69 to 6.34), no single out-performing algorithm was identified: FastSurfer performed best in VS, QuickNat in DICE and average HD, and Hippodeep in HD. Segmentation performance was significantly lower for ipsilesional segmentation, with a decrease in performance as a function of lesion size due to the pathology-based domain shift. Only QuickNat showed a more robust performance in volumetric similarity. Even though there are many pre-trained segmentation methods, it is important to be aware of the possible decrease in performance for the segmentation results on the lesion side due to the pathology-based domain shift. The segmentation algorithm should be selected based on the research question and the evaluation parameter needed. More research is needed to improve current hippocampal segmentation methods.

Joint EANM/SNMMI/ESTRO practice recommendations for the use of 2-[18F]FDG PET/CT external beam radiation treatment planning in lung cancer V1.0

PURPOSE

2-[18F]FDG PET/CT is of utmost importance for radiation treatment (RT) planning and response monitoring in lung cancer patients, in both non-small and small cell lung cancer (NSCLC and SCLC). This topic has been addressed in guidelines composed by experts within the field of radiation oncology. However, up to present, there is no procedural guideline on this subject, with involvement of the nuclear medicine societies.

METHODS

A literature review was performed, followed by a discussion between a multidisciplinary team of experts in the different fields involved in the RT planning of lung cancer, in order to guide clinical management. The project was led by experts of the two nuclear medicine societies (EANM and SNMMI) and radiation oncology (ESTRO).

RESULTS AND CONCLUSION

This guideline results from a joint and dynamic collaboration between the relevant disciplines for this topic. It provides a worldwide, state of the art, and multidisciplinary guide to 2-[18F]FDG PET/CT RT planning in NSCLC and SCLC. These practical recommendations describe applicable updates for existing clinical practices, highlight potential flaws, and provide solutions to overcome these as well. Finally, the recent developments considered for future application are also reviewed.

Interobserver Agreement on Automated Metabolic Tumor Volume Measurements of Deauville Score 4 and 5 Lesions at Interim 18F-FDG PET in Diffuse Large B-Cell Lymphoma

Metabolic tumor volume (MTV) on interim PET (I-PET) is a potential prognostic biomarker for diffuse large B-cell lymphoma (DLBCL). Implementation of MTV on I-PET requires a consensus on which semiautomated segmentation method delineates lesions most successfully with least user interaction. Methods used for baseline PET are not necessarily optimal for I-PET because of lower lesional SUVs at I-PET. Therefore, we aimed to evaluate which method provides the best delineation quality for Deauville score (DS) 4-5 DLBCL lesions on I-PET at the best interobserver agreement on delineation quality and, second, to assess the effect of lesional SUVmax on delineation quality and performance agreement. Methods: DS 4-5 lesions from 45 I-PET scans were delineated using 6 semiautomated methods: a fixed SUV threshold of 2.5 g/cm3, a fixed SUV threshold of 4.0 g/cm3, an adaptive threshold corrected for source-to-local background activity contrast at 50% of the SUVpeak, 41% of SUVmax per lesion, a majority vote including voxels detected by at least 2 methods, and a majority vote including voxels detected by at least 3 methods (MV3). Delineation quality per MTV was rated by 3 independent observers as acceptable or nonacceptable. For each method, observer scores on delineation quality, specific agreement, and MTV were assessed for all lesions and per category of lesional SUVmax (<5, 5-10, >10). Results: In 60 DS 4-5 lesions on I-PET, MV3 performed best, with acceptable delineation in 90% of lesions and a positive agreement of 93%. Delineation quality scores and agreement per method strongly depended on lesional SUV: the best delineation quality scores were obtained using MV3 in lesions with an SUVmax of less than 10 and using SUV4.0 in more 18F-FDG-avid lesions. Consequently, overall delineation quality and positive agreement improved by applying the most preferred method per SUV category instead of using MV3 as the single best method. The MV3- and SUV4.0-derived MTVs of lesions with an SUVmax of more than 10 were comparable after exclusion of visually failed MV3 contouring. For lesions with an SUVmax of less than 10, MTVs using different methods correlated poorly. Conclusion: On I-PET, MV3 performed best and provided the highest interobserver agreement regarding acceptable delineations of DS 4-5 DLBCL lesions. However, delineation-method preference strongly depended on lesional SUV. Therefore, we suggest exploration of an approach that identifies the optimal delineation method per lesion as a function of tumor 18F-FDG uptake characteristics, that is, SUVmax.

Esophagus Segmentation in CT Images via Spatial Attention Network and STAPLE Algorithm

One essential step in radiotherapy treatment planning is the organ at risk of segmentation in Computed Tomography (CT). Many recent studies have focused on several organs such as the lung, heart, esophagus, trachea, liver, aorta, kidney, and prostate. However, among the above organs, the esophagus is one of the most difficult organs to segment because of its small size, ambiguous boundary, and very low contrast in CT images. To address these challenges, we propose a fully automated framework for the esophagus segmentation from CT images. The proposed method is based on the processing of slice images from the original three-dimensional (3D) image so that our method does not require large computational resources. We employ the spatial attention mechanism with the atrous spatial pyramid pooling module to locate the esophagus effectively, which enhances the segmentation performance. To optimize our model, we use group normalization because the computation is independent of batch sizes, and its performance is stable. We also used the simultaneous truth and performance level estimation (STAPLE) algorithm to reach robust results for segmentation. Firstly, our model was trained by k-fold cross-validation. And then, the candidate labels generated by each fold were combined by using the STAPLE algorithm. And as a result, Dice and Hausdorff Distance scores have an improvement when applying this algorithm to our segmentation results. Our method was evaluated on SegTHOR and StructSeg 2019 datasets, and the experiment shows that our method outperforms the state-of-the-art methods in esophagus segmentation. Our approach shows a promising result in esophagus segmentation, which is still challenging in medical analyses.

A Bayesian approach to tissue-fraction estimation for oncological PET segmentation

Tumor segmentation in oncological PET is challenging, a major reason being the partial-volume effects (PVEs) that arise due to low system resolution and finite voxel size. The latter results in tissue-fraction effects (TFEs), i.e. voxels contain a mixture of tissue classes. Conventional segmentation methods are typically designed to assign each image voxel as belonging to a certain tissue class. Thus, these methods are inherently limited in modeling TFEs. To address the challenge of accounting for PVEs, and in particular, TFEs, we propose a Bayesian approach to tissue-fraction estimation for oncological PET segmentation. Specifically, this Bayesian approach estimates the posterior mean of the fractional volume that the tumor occupies within each image voxel. The proposed method, implemented using a deep-learning-based technique, was first evaluated using clinically realistic 2D simulation studies with known ground truth, in the context of segmenting the primary tumor in PET images of patients with lung cancer. The evaluation studies demonstrated that the method accurately estimated the tumor-fraction areas and significantly outperformed widely used conventional PET segmentation methods, including a U-net-based method, on the task of segmenting the tumor. In addition, the proposed method was relatively insensitive to PVEs and yielded reliable tumor segmentation for different clinical-scanner configurations. The method was then evaluated using clinical images of patients with stage IIB/III non-small cell lung cancer from ACRIN 6668/RTOG 0235 multi-center clinical trial. Here, the results showed that the proposed method significantly outperformed all other considered methods and yielded accurate tumor segmentation on patient images with Dice similarity coefficient (DSC) of 0.82 (95% CI: 0.78, 0.86). In particular, the method accurately segmented relatively small tumors, yielding a high DSC of 0.77 for the smallest segmented cross-section of 1.30 cm². Overall, this study demonstrates the efficacy of the proposed method to accurately segment tumors in PET images.

MRI Diffusion-Weighted Imaging to Measure Infarct Volume: Assessment of Manual Segmentation Variability

BACKGROUND AND PURPOSE

Manual segmentation of infarct volume on follow-up MRI diffusion-weighted imaging (MRI-DWI) is considered the gold standard but is prone to rater variability. We assess the variability of manual segmentations of MRI-DWI infarct volume.

METHODS

Consecutive patients (May 2018 to May 2019) with the anterior circulation stroke and endovascularly treated were enrolled. All patients underwent 24- to 32-hour follow-up MRI. Three users manually segmented DWI infarct volumes slice by slice twice. The reference standard of DWI infarct volume was generated by the STAPLE algorithm. Intra- and interrater reliability was evaluated using the intraclass correlation coefficient (ICC) by comparing manual segmentations with the reference standard. Spatial measurements were evaluated using metrics of the Dice similarity coefficient (DSC). Volumetric measurements were compared using the lesion volume.

RESULTS

The dataset consisted of 44 patients, mean (SD) age was 70.1 years (±10.3), 43% were women, and median baseline NIHSS score was 16. Among three users, the mean DSC for MRI-DWI infarct volume segmentations ranged from 80.6% ± 11.7% to 88.6% ± 7.5%, and the mean absolute volume difference was 2.8 ± 6.8 to 13.0 ± 14.0 ml. Interrater ICC among the users for DSC and infarct volume was .86 (95% confidence interval [95% CI]: .78-.91) and .997 (95% CI: .995-.998). Intrarater ICC for the three users was .83 (95% CI: .69-.93), .84 (95% CI: .72-.91), and .80 (95% CI: .64-.89) for DSC, and .99 (95% CI: .987-.996), .991 (95% CI: .983-.995), and .996 (95% CI: .993-.998) for infarct volume.

CONCLUSIONS

Manual segmentation of infarct volume on follow-up MRI-DWI shows excellent agreement and good spatial overlap with the reference standard, suggesting its usefulness for measuring infarct volume on 24- to 32-hour MRI-DWI.

Spatial agreement of demineralized areas in quantitative light-induced fluorescence images and digital photographs

OBJECTIVES

Previous work has shown qualitatively that detection of demineralized tooth areas (white spot lesions, WSLs) is more reliable in digital photographs (DP) as in quantitative light-induced fluorescence (QLF) images. Based on non-rigid, multimodal image registration, we now quantitatively compare manual and automatic markings in both modalities.

METHODS

After braces removal, pairs of DP and QLF were acquired from 124 teeth of 31 patients. Three experienced raters marked the WSL on both DP and QLF images, each of which was presented twice in randomized order. For each tooth and each modality, a ground truth (GT) was established using the simultaneous truth and performance level estimation algorithm on the total of six manual markings per image. DP and QLF image pairs were spatially registered, by aligning the outline of the tooth area in DPs to that of the corresponding tooth area in QLF. Between all pairs of markings for all teeth, position and size were compared quantitatively by the Dice coefficient and the novel coefficient of inclusion.

RESULTS

Our hypotheses: (i) the clinical inspection supported by DP is more sensitive to WSL as that by QLF, disregarding whether the automatic analysis or the experts' manual assessment of QLF is applied, and (ii) detected lesions in QLF are included in those of DP, were confirmed and not confirmed, respectively.

CONCLUSION

DP and QLF are valuable methods to detect WSL in demineralized teeth. Combining both modalities can provide additional information on early lesion assessment.

The first MICCAI challenge on PET tumor segmentation

INTRODUCTION

Automatic functional volume segmentation in PET images is a challenge that has been addressed using a large array of methods. A major limitation for the field has been the lack of a benchmark dataset that would allow direct comparison of the results in the various publications. In the present work, we describe a comparison of recent methods on a large dataset following recommendations by the American Association of Physicists in Medicine (AAPM) task group (TG) 211, which was carried out within a MICCAI (Medical Image Computing and Computer Assisted Intervention) challenge.

MATERIALS AND METHODS

Organization and funding was provided by France Life Imaging (FLI). A dataset of 176 images combining simulated, phantom and clinical images was assembled. A website allowed the participants to register and download training data (n = 19). Challengers then submitted encapsulated pipelines on an online platform that autonomously ran the algorithms on the testing data (n = 157) and evaluated the results. The methods were ranked according to the arithmetic mean of sensitivity and positive predictive value.

RESULTS

Sixteen teams registered but only four provided manuscripts and pipeline(s) for a total of 10 methods. In addition, results using two thresholds and the Fuzzy Locally Adaptive Bayesian (FLAB) were generated. All competing methods except one performed with median accuracy above 0.8. The method with the highest score was the convolutional neural network-based segmentation, which significantly outperformed 9 out of 12 of the other methods, but not the improved K-Means, Gaussian Model Mixture and Fuzzy C-Means methods.

CONCLUSION

The most rigorous comparative study of PET segmentation algorithms to date was carried out using a dataset that is the largest used in such studies so far. The hierarchy amongst the methods in terms of accuracy did not depend strongly on the subset of datasets or the metrics (or combination of metrics). All the methods submitted by the challengers except one demonstrated good performance with median accuracy scores above 0.8.

Quantitative light-induced fluorescence images and digital photographs - Reproducibility of manually marked demineralisations

OBJECTIVE

Hard tooth tissue demineralisation is an undesirable side effect of orthodontic treatment with fixed appliances. Whereas both clinically and in digital photographs (DP), demineralisations appear as white spot lesions, WSLs appear as dark areas when quantitative light-induced fluorescence (QLF) imaging is used. This study aims at comparing the reproducibility of the detection of decalcified tooth areas in DP and QLF.

MATERIALS AND METHODS

DP and QLF pairs were acquired from 139 teeth of 32 patients after braces removal. Three raters manually marked the decalcified area on both DP and QLF images. The markings were repeated after 2 weeks. A ground truth was estimated for each tooth and modality using the simultaneous truth and performance level estimation (STAPLE) algorithm. The Dice coefficients (DC) of each rater marking to the ground truth were calculated for all teeth and modalities to quantify the spatial agreement. A three-way repeated measures analysis of variance (ANOVA) was used to compare the means of the DCs for both modalities ([Formula: see text]). Intra-observer and intercycle variabilities were assessed comparing the means across the raters and the cycles for both modalities.

RESULTS

ANOVA revealed a statistical significant difference between the modalities [[Formula: see text], [Formula: see text]]. The standard deviation of the DC for the photographs are lower than those for the QLF images. Intra-observer and intercycle differences are rather small as compared to the intermodality differences.

CONCLUSIONS

The results indicate a higher spatial reproducibility in identifying a decalcified area on a tooth surface using visual inspection of DP rather than QLF images.

Deterioration of R-Wave Detection in Pathology and Noise: A Comprehensive Analysis Using Simultaneous Truth and Performance Level Estimation

OBJECTIVE

For long-term electrocardiography (ECG) recordings, accurate R-wave detection is essential. Several algorithms have been proposed but not yet compared on large, noisy, or pathological data, since manual ground-truth establishment is impossible on such large data.

METHODS

We apply the simultaneous truth and performance level estimation (STAPLE) method to ECG signals comparing nine R-wave detectors: Pan and Tompkins (1985), Chernenko (2007), Arzeno et al. (2008), Manikandan et al. (2012), Lentini et al. (2013), Sartor et al. (2014), Liu et al. (2014), Arteaga-Falconi et al. (2015), and Khamis et al. (2016). Experiments are performed on the MIT-BIH database, TELE database, PTB database, and 24/7 Holter recordings of 60 multimorbid subjects.

RESULTS

Existing approaches on R-wave detection perform excellently on healthy subjects (F-measure above 99% for most methods), but performance drops to a range of F = 90.10% (Khamis et al.) to F = 30.10% (Chernenko) when analyzing the 37 million R-waves of multimorbid subjects. STAPLE improves existing approaches (ΔF = 0.04 for the MIT-BIH database and ΔF = 0.95 for the TELE database) and yields a relative (not absolute) scale to compare algorithms' performances.

CONCLUSION

More robust R-wave detection methods or flexible combinations are required to analyze continuous data captured from pathological subjects or that is recorded with dropouts and noise.

SIGNIFICANCE

STAPLE algorithm has been adopted from image to signal analysis to compare algorithms on large, incomplete, and noisy data without manual ground truth. Existing approaches on R-wave detection weakly perform on such data.

ATLAAS: an automatic decision tree-based learning algorithm for advanced image segmentation in positron emission tomography

Accurate and reliable tumour delineation on positron emission tomography (PET) is crucial for radiotherapy treatment planning. PET automatic segmentation (PET-AS) eliminates intra- and interobserver variability, but there is currently no consensus on the optimal method to use, as different algorithms appear to perform better for different types of tumours. This work aimed to develop a predictive segmentation model, trained to automatically select and apply the best PET-AS method, according to the tumour characteristics. ATLAAS, the automatic decision tree-based learning algorithm for advanced segmentation is based on supervised machine learning using decision trees. The model includes nine PET-AS methods and was trained on a 100 PET scans with known true contour. A decision tree was built for each PET-AS algorithm to predict its accuracy, quantified using the Dice similarity coefficient (DSC), according to the tumour volume, tumour peak to background SUV ratio and a regional texture metric. The performance of ATLAAS was evaluated for 85 PET scans obtained from fillable and printed subresolution sandwich phantoms. ATLAAS showed excellent accuracy across a wide range of phantom data and predicted the best or near-best segmentation algorithm in 93% of cases. ATLAAS outperformed all single PET-AS methods on fillable phantom data with a DSC of 0.881, while the DSC for H&N phantom data was 0.819. DSCs higher than 0.650 were achieved in all cases. ATLAAS is an advanced automatic image segmentation algorithm based on decision tree predictive modelling, which can be trained on images with known true contour, to predict the best PET-AS method when the true contour is unknown. ATLAAS provides robust and accurate image segmentation with potential applications to radiation oncology.