METhodological RadiomICs Score (METRICS): a quality scoring tool for radiomics research endorsed by EuSoMII

PURPOSE

To propose a new quality scoring tool, METhodological RadiomICs Score (METRICS), to assess and improve research quality of radiomics studies.

METHODS

We conducted an online modified Delphi study with a group of international experts. It was performed in three consecutive stages: Stage#1, item preparation; Stage#2, panel discussion among EuSoMII Auditing Group members to identify the items to be voted; and Stage#3, four rounds of the modified Delphi exercise by panelists to determine the items eligible for the METRICS and their weights. The consensus threshold was 75%. Based on the median ranks derived from expert panel opinion and their rank-sum based conversion to importance scores, the category and item weights were calculated.

RESULT

In total, 59 panelists from 19 countries participated in selection and ranking of the items and categories. Final METRICS tool included 30 items within 9 categories. According to their weights, the categories were in descending order of importance: study design, imaging data, image processing and feature extraction, metrics and comparison, testing, feature processing, preparation for modeling, segmentation, and open science. A web application and a repository were developed to streamline the calculation of the METRICS score and to collect feedback from the radiomics community.

CONCLUSION

In this work, we developed a scoring tool for assessing the methodological quality of the radiomics research, with a large international panel and a modified Delphi protocol. With its conditional format to cover methodological variations, it provides a well-constructed framework for the key methodological concepts to assess the quality of radiomic research papers.

CRITICAL RELEVANCE STATEMENT

A quality assessment tool, METhodological RadiomICs Score (METRICS), is made available by a large group of international domain experts, with transparent methodology, aiming at evaluating and improving research quality in radiomics and machine learning.

KEY POINTS

• A methodological scoring tool, METRICS, was developed for assessing the quality of radiomics research, with a large international expert panel and a modified Delphi protocol. • The proposed scoring tool presents expert opinion-based importance weights of categories and items with a transparent methodology for the first time. • METRICS accounts for varying use cases, from handcrafted radiomics to entirely deep learning-based pipelines. • A web application has been developed to help with the calculation of the METRICS score ( https://metricsscore.github.io/metrics/METRICS.html ) and a repository created to collect feedback from the radiomics community ( https://github.com/metricsscore/metrics ).

Getting Cartilage Thickness Measurements Right: A Systematic Inter-Method Comparison Using MRI Data from the Osteoarthritis Initiative

OBJECTIVE

Magnetic resonance imaging is the standard imaging modality to assess articular cartilage. As the imaging surrogate of degenerative joint disease, cartilage thickness is commonly quantified after tissue segmentation. In lack of a standard method, this study systematically compared five methods for automatic cartilage thickness measurements across the knee joint and as a function of region and sub-region: 3D mesh normals (3D-MN), 3D nearest neighbors (3D-NN), 3D ray tracing (3D-RT), 2D centerline normals (2D-CN), and 2D surface normals (2D-SN).

DESIGN

Based on the manually segmented femoral and tibial cartilage of 507 human knee joints, mean cartilage thickness was computed for the entire femorotibial joint, 4 joint regions, and 20 subregions using these methods. Inter-method comparisons of mean cartilage thickness and computation times were performed by one-way analysis of variance (ANOVA), Bland-Altman analyses and Lin's concordance correlation coefficient (CCC).

RESULTS

Mean inter-method differences in cartilage thickness were significant in nearly all subregions (P < 0.001). By trend, mean differences were smallest between 3D-MN and 2D-SN in most (sub)regions, which is also reflected by highest quantitative inter-method agreement and CCCs. 3D-RT was prone to severe overestimation of up to 2.5 mm. 3D-MN, 3D-NN, and 2D-SN required mean processing times of ≤5.3 s per joint and were thus similarly efficient, whereas the time demand of 2D-CN and 3D-RT was much larger at 133 ± 29 and 351 ± 10 s per joint (P < 0.001).

CONCLUSIONS

In automatic cartilage thickness determination, quantification accuracy and computational burden are largely affected by the underlying method. Mesh and surface normals or nearest neighbor searches should be used because they accurately capture variable geometries while being time-efficient.

Mitigation of noise-induced bias of PET radiomic features

Introduction

One major challenge in PET radiomics is its sensitivity to noise. Low signal-to-noise ratio (SNR) affects not only the precision but also the accuracy of quantitative metrics extracted from the images resulting in noise-induced bias. This phantom study aims to identify the radiomic features that are robust to noise in terms of precision and accuracy and to explore some methods that might help to correct noise-induced bias.

Methods

A phantom containing three ¹⁸F-FDG filled 3D printed inserts, reflecting heterogeneous tracer uptake and realistic tumor shapes, was used in the study. The three different phantom inserts were filled and scanned with three different tumor-to-background ratios, simulating a total of nine different tumors. From the 40-minute list-mode data, ten frames each for 5 s, 10 s, 30 s, and 120 s frame duration were reconstructed to generate images with different noise levels. Under these noise conditions, the precision and accuracy of the radiomic features were analyzed using intraclass correlation coefficient (ICC) and similarity distance metric (SDM) respectively. Based on the ICC and SDM values, the radiomic features were categorized into four groups: poor, moderate, good, and excellent precision and accuracy. A “difference image” created by subtracting two statistically equivalent replicate images was used to develop a model to correct the noise-induced bias. Several regression methods (e.g., linear, exponential, sigmoid, and power-law) were tested. The best fitting model was chosen based on Akaike information criteria.

Results

Several radiomic features derived from low SNR images have high repeatability, with 68% of radiomic features having ICC ≥ 0.9 for images with a frame duration of 5 s. However, most features show a systematic bias that correlates with the increase in noise level. Out of 143 features with noise-induced bias, the SDM values were improved based on a regression model (53 features to excellent and 67 to good) indicating that the noise-induced bias of these features can be, at least partially, corrected.

Conclusion

To have a predictive value, radiomic features should reflect tumor characteristics and be minimally affected by noise. The present study has shown that it is possible to correct for noise-induced bias, at least in a subset of the features, using a regression model based on the local image noise estimates.

Convolutional neural networks for automatic image quality control and EARL compliance of PET images

Background

Machine learning studies require a large number of images often obtained on different PET scanners. When merging these images, the use of harmonized images following EARL-standards is essential. However, when including retrospective images, EARL accreditation might not have been in place. The aim of this study was to develop a convolutional neural network (CNN) that can identify retrospectively if an image is EARL compliant and if it is meeting older or newer EARL-standards.

Materials and methods

96 PET images acquired on three PET/CT systems were included in the study. All images were reconstructed with the locally clinically preferred, EARL1, and EARL2 compliant reconstruction protocols. After image pre-processing, one CNN was trained to separate clinical and EARL compliant reconstructions. A second CNN was optimized to identify EARL1 and EARL2 compliant images. The accuracy of both CNNs was assessed using fivefold cross-validation. The CNNs were validated on 24 images acquired on a PET scanner not included in the training data. To assess the impact of image noise on the CNN decision, the 24 images were reconstructed with different scan durations.

Results

In the cross-validation, the first CNN classified all images correctly. When identifying EARL1 and EARL2 compliant images, the second CNN identified 100% EARL1 compliant and 85% EARL2 compliant images correctly. The accuracy in the independent dataset was comparable to the cross-validation accuracy. The scan duration had almost no impact on the results.

Conclusion

The two CNNs trained in this study can be used to retrospectively include images in a multi-center setting by, e.g., adding additional smoothing. This method is especially important for machine learning studies where the harmonization of images from different PET systems is essential.

Supplementary Information

The online version contains supplementary material available at 10.1186/s40658-022-00468-w.

Noise sensitivity of 89Zr-Immuno-PET radiomics based on count-reduced clinical images

PURPOSE

Low photon count in ⁸⁹Zr-Immuno-PET results in images with a low signal-to-noise ratio (SNR). Since PET radiomics are sensitive to noise, this study focuses on the impact of noise on radiomic features from ⁸⁹Zr-Immuno-PET clinical images. We hypothesise that ⁸⁹Zr-Immuno-PET derived radiomic features have: (1) noise-induced variability affecting their precision and (2) noise-induced bias affecting their accuracy. This study aims to identify those features that are not or only minimally affected by noise in terms of precision and accuracy.

METHODS

Count-split ⁸⁹Zr-Immuno-PET patient scans from previous studies with three different ⁸⁹Zr-labelled monoclonal antibodies were used to extract radiomic features at 50% (S50p) and 25% (S25p) of their original counts. Tumour lesions were manually delineated on the original full-count ⁸⁹Zr-Immuno-PET scans. Noise-induced variability and bias were assessed using intraclass correlation coefficient (ICC) and similarity distance metric (SDM), respectively. Based on the ICC and SDM values, the radiomic features were categorised as having poor [0, 0.5), moderate [0.5, 0.75), good [0.75, 0.9), or excellent [0.9, 1] precision and accuracy. The number of features classified into these categories was compared between the S50p and S25p images using Fisher's exact test. All p values < 0.01 were considered statistically significant.

RESULTS

For S50p, a total of 92% and 90% features were classified as having good or excellent ICC and SDM respectively, while for S25p, these decreased to 81% and 31%. In total, 148 features (31%) showed robustness to noise with good or moderate ICC and SDM in both S50p and S25p. The number of features classified into the four ICC and SDM categories between S50p and S25p was significantly different statistically.

CONCLUSION

Several radiomic features derived from low SNR ⁸⁹Zr-Immuno-PET images exhibit noise-induced variability and/or bias. However, 196 features (43%) that show minimal noise-induced variability and bias in S50p images have been identified. These features are less affected by noise and are, therefore, suitable candidates to be further studied as prognostic and predictive quantitative biomarkers in ⁸⁹Zr-Immuno-PET studies.

Segmentation Uncertainty Estimation as a Sanity Check for Image Biomarker Studies

Problem. Image biomarker analysis, also known as radiomics, is a tool for tissue characterization and treatment prognosis that relies on routinely acquired clinical images and delineations. Due to the uncertainty in image acquisition, processing, and segmentation (delineation) protocols, radiomics often lack reproducibility. Radiomics harmonization techniques have been proposed as a solution to reduce these sources of uncertainty and/or their influence on the prognostic model performance. A relevant question is how to estimate the protocol-induced uncertainty of a specific image biomarker, what the effect is on the model performance, and how to optimize the model given the uncertainty. Methods. Two non-small cell lung cancer (NSCLC) cohorts, composed of 421 and 240 patients, respectively, were used for training and testing. Per patient, a Monte Carlo algorithm was used to generate three hundred synthetic contours with a surface dice tolerance measure of less than 1.18 mm with respect to the original GTV. These contours were subsequently used to derive 104 radiomic features, which were ranked on their relative sensitivity to contour perturbation, expressed in the parameter η. The top four (low η) and the bottom four (high η) features were selected for two models based on the Cox proportional hazards model. To investigate the influence of segmentation uncertainty on the prognostic model, we trained and tested the setup in 5000 augmented realizations (using a Monte Carlo sampling method); the log-rank test was used to assess the stratification performance and stability of segmentation uncertainty. Results. Although both low and high η setup showed significant testing set log-rank p-values (p = 0.01) in the original GTV delineations (without segmentation uncertainty introduced), in the model with high uncertainty, to effect ratio, only around 30% of the augmented realizations resulted in model performance with p < 0.05 in the test set. In contrast, the low η setup performed with a log-rank p < 0.05 in 90% of the augmented realizations. Moreover, the high η setup classification was uncertain in its predictions for 50% of the subjects in the testing set (for 80% agreement rate), whereas the low η setup was uncertain only in 10% of the cases. Discussion. Estimating image biomarker model performance based only on the original GTV segmentation, without considering segmentation, uncertainty may be deceiving. The model might result in a significant stratification performance, but can be unstable for delineation variations, which are inherent to manual segmentation. Simulating segmentation uncertainty using the method described allows for more stable image biomarker estimation, selection, and model development. The segmentation uncertainty estimation method described here is universal and can be extended to estimate other protocol uncertainties (such as image acquisition and pre-processing).

A systematic review and quality of reporting checklist for repeatability and reproducibility of radiomic features

•

Main factors impacting feature stability: Image acquisition, reconstruction, tumor segmentation, and interpolation.

•

Textural features are less robust than morphological or statistical features.

•

A checklist is provided including items that should be reported in a radiomic study.

Purpose

Although quantitative image biomarkers (radiomics) show promising value for cancer diagnosis, prognosis, and treatment assessment, these biomarkers still lack reproducibility. In this systematic review, we aimed to assess the progress in radiomics reproducibility and repeatability in the recent years.

Methods and materials

Four hundred fifty-one abstracts were retrieved according to the original PubMed search pattern with the publication dates ranging from 2017/05/01 to 2020/12/01. Each abstract including the keywords was independently screened by four observers. Forty-two full-text articles were selected for further analysis. Patient population data, radiomic feature classes, feature extraction software, image preprocessing, and reproducibility results were extracted from each article. To support the community with a standardized reporting strategy, we propose a specific reporting checklist to evaluate the feasibility to reproduce each study.

Results

Many studies continue to under-report essential reproducibility information: all but one clinical and all but two phantom studies missed to report at least one important item reporting image acquisition. The studies included in this review indicate that all radiomic features are sensitive to image acquisition, reconstruction, tumor segmentation, and interpolation. However, the amount of sensitivity is feature dependent, for instance, textural features were, in general, less robust than statistical features.

Conclusions

Radiomics repeatability, reproducibility, and reporting quality can substantially be improved regarding feature extraction software and settings, image preprocessing and acquisition, cutoff values for stable feature selection. Our proposed radiomics reporting checklist can serve to simplify and improve the reporting and, eventually, guarantee the possibility to fully replicate and validate radiomic studies.

Plausibility and redundancy analysis to select FDG-PET textural features in non-small cell lung cancer

Background

Radiomics refers to the extraction of a large number of image biomarker describing the tumor phenotype displayed in a medical image. Extracted from positron emission tomography (PET) images, radiomics showed diagnostic and prognostic value for several cancer types. However, a large number of radiomic features are nonreproducible or highly correlated with conventional PET metrics. Moreover, radiomic features used in the clinic should yield relevant information about tumor texture. In this study, we propose a framework to identify technical and clinical meaningful features and exemplify our results using a PET non‐small cell lung cancer (NSCLC) dataset.

Materials and methods

The proposed selection procedure consists of several steps. A priori, we only include features that were found to be reproducible in a multicenter setting. Next, we apply a voxel randomization step to identify features that reflect actual textural information, that is, that yield in 90% of the patient scans a value significantly different from random texture. Finally, the remaining features were correlated with standard PET metrics to further remove redundancy with common PET metrics. The selection procedure was performed for different volume ranges, that is, excluding lesions with smaller volumes in order to assess the effect of tumor size on the results. To exemplify our procedure, the selected features were used to predict 1‐yr survival in a dataset of 150 NSCLC patients. A predictive model was built using volume as predictive factor for smaller, and one of the selected features as predictive factor for bigger lesions. The prediction accuracy of the both models were compared with the prediction accuracy of volume.

Results

The number of selected features depended on the lesion size included in the analysis. When including the whole dataset, from 19 features reflecting actual texture only two were found to be not strongly correlated with conventional PET metrics. When excluding lesions smaller than 11.49 and 33.10 mL (25 and 50 percentile of the dataset), four out of 27 features and 13 out of 29 features remained after eliminating features highly correlated with standard PET metrics. When excluding lesions smaller than 103.9 mL (75 percentile), 33 out of 53 features remained. For larger lesions, some of these features outperformed volume in terms of classification accuracy (increase of 4–10%). The combination of using volume as predictor for smaller and one of the selected features for larger lesions also improved the accuracy when compared with volume only (increase from 72% to 76%).

Conclusion

When performing radiomic analysis for smaller lesions, it should be first carefully investigated if a textural feature reflects actual heterogeneity information. Next, verification of the absence of correlation with all conventional PET metrics is essential in order to assess the additional value of radiomic features. Radiomic analysis with lesions larger than 11.4 mL might give additional information to conventional metrics while at the same time reflecting actual tumor texture. Using a combination of volume and one of the selected features for prediction yields promise to increase accuracy and reliability of a radiomic model.

Repeatability of two semi-automatic artificial intelligence approaches for tumor segmentation in PET

Background

Positron emission tomography (PET) is routinely used for cancer staging and treatment follow-up. Metabolic active tumor volume (MATV) as well as total MATV (TMATV—including primary tumor, lymph nodes and metastasis) and/or total lesion glycolysis derived from PET images have been identified as prognostic factor or for the evaluation of treatment efficacy in cancer patients. To this end, a segmentation approach with high precision and repeatability is important. However, the implementation of a repeatable and accurate segmentation algorithm remains an ongoing challenge.

Methods

In this study, we compare two semi-automatic artificial intelligence (AI)-based segmentation methods with conventional semi-automatic segmentation approaches in terms of repeatability. One segmentation approach is based on a textural feature (TF) segmentation approach designed for accurate and repeatable segmentation of primary tumors and metastasis. Moreover, a convolutional neural network (CNN) is trained. The algorithms are trained, validated and tested using a lung cancer PET dataset. The segmentation accuracy of both segmentation approaches is compared using the Jaccard coefficient (JC). Additionally, the approaches are externally tested on a fully independent test–retest dataset. The repeatability of the methods is compared with those of two majority vote (MV2, MV3) approaches, 41%SUV_MAX, and a SUV > 4 segmentation (SUV4). Repeatability is assessed with test–retest coefficients (TRT%) and intraclass correlation coefficient (ICC). An ICC > 0.9 was regarded as representing excellent repeatability.

Results

The accuracy of the segmentations with the reference segmentation was good (JC median TF: 0.7, CNN: 0.73). Both segmentation approaches outperformed most other conventional segmentation methods in terms of test–retest coefficient (TRT% mean: TF: 13.0%, CNN: 13.9%, MV2: 14.1%, MV3: 28.1%, 41%SUV_MAX: 28.1%, SUV4: 18.1%) and ICC (TF: 0.98, MV2: 0.97, CNN: 0.99, MV3: 0.73, SUV4: 0.81, and 41%SUV_MAX: 0.68).

Conclusion

The semi-automatic AI-based segmentation approaches used in this study provided better repeatability than conventional segmentation approaches. Moreover, both algorithms lead to accurate segmentations for both primary tumors as well as metastasis and are therefore good candidates for PET tumor segmentation.

Predictive value of quantitative 18F-FDG-PET radiomics analysis in patients with head and neck squamous cell carcinoma

BACKGROUND

Radiomics is aimed at image-based tumor phenotyping, enabling application within clinical-decision-support-systems to improve diagnostic accuracy and allow for personalized treatment. The purpose was to identify predictive 18-fluor-fluoro-2-deoxyglucose (¹⁸F-FDG) positron-emission tomography (PET) radiomic features to predict recurrence, distant metastasis, and overall survival in patients with head and neck squamous cell carcinoma treated with chemoradiotherapy.

METHODS

Between 2012 and 2018, 103 retrospectively (training cohort) and 71 consecutively included patients (validation cohort) underwent ¹⁸F-FDG-PET/CT imaging. The 434 extracted radiomic features were subjected, after redundancy filtering, to a projection resulting in outcome-independent meta-features (factors). Correlations between clinical, first-order ¹⁸F-FDG-PET parameters (e.g., SUVmean), and factors were assessed. Factors were combined with ¹⁸F-FDG-PET and clinical parameters in a multivariable survival regression and validated. A clinically applicable risk-stratification was constructed for patients' outcome.

RESULTS

Based on 124 retained radiomic features from 103 patients, 8 factors were constructed. Recurrence prediction was significantly most accurate by combining HPV-status, SUVmean, SUVpeak, factor 3 (histogram gradient and long-run-low-grey-level-emphasis), factor 4 (volume-difference, coarseness, and grey-level-non-uniformity), and factor 6 (histogram variation coefficient) (CI = 0.645). Distant metastasis prediction was most accurate assessing metabolic-active tumor volume (MATV)(CI = 0.627). Overall survival prediction was most accurate using HPV-status, SUVmean, SUVmax, factor 1 (least-axis-length, non-uniformity, high-dependence-of-high grey-levels), and factor 5 (aspherity, major-axis-length, inversed-compactness and, inversed-flatness) (CI = 0.764).

CONCLUSIONS

Combining HPV-status, first-order ¹⁸F-FDG-PET parameters, and complementary radiomic factors was most accurate for time-to-event prediction. Predictive phenotype-specific tumor characteristics and interactions might be captured and retained using radiomic factors, which allows for personalized risk stratification and optimizing personalized cancer care.

TRIAL REGISTRATION

Trial NL3946 (NTR4111), local ethics commission reference: Prediction 2013.191 and 2016.498. Registered 7 August 2013, https://www.trialregister.nl/trial/3946.

Machine learning-based analysis of [18F]DCFPyL PET radiomics for risk stratification in primary prostate cancer

PURPOSE

Quantitative prostate-specific membrane antigen (PSMA) PET analysis may provide for non-invasive and objective risk stratification of primary prostate cancer (PCa) patients. We determined the ability of machine learning-based analysis of quantitative [¹⁸F]DCFPyL PET metrics to predict metastatic disease or high-risk pathological tumor features.

METHODS

In a prospective cohort study, 76 patients with intermediate- to high-risk PCa scheduled for robot-assisted radical prostatectomy with extended pelvic lymph node dissection underwent pre-operative [¹⁸F]DCFPyL PET-CT. Primary tumors were delineated using 50-70% peak isocontour thresholds on images with and without partial-volume correction (PVC). Four hundred and eighty standardized radiomic features were extracted per tumor. Random forest models were trained to predict lymph node involvement (LNI), presence of any metastasis, Gleason score ≥ 8, and presence of extracapsular extension (ECE). For comparison, models were also trained using standard PET features (SUVs, volume, total PSMA uptake). Model performance was validated using 50 times repeated 5-fold cross-validation yielding the mean receiver-operator characteristic curve AUC.

RESULTS

The radiomics-based machine learning models predicted LNI (AUC 0.86 ± 0.15, p < 0.01), nodal or distant metastasis (AUC 0.86 ± 0.14, p < 0.01), Gleason score (0.81 ± 0.16, p < 0.01), and ECE (0.76 ± 0.12, p < 0.01). The highest AUCs reached using standard PET metrics were lower than those of radiomics-based models. For LNI and metastasis prediction, PVC and a higher delineation threshold improved model stability. Machine learning pre-processing methods had a minor impact on model performance.

CONCLUSION

Machine learning-based analysis of quantitative [¹⁸F]DCFPyL PET metrics can predict LNI and high-risk pathological tumor features in primary PCa patients. These findings indicate that PSMA expression detected on PET is related to both primary tumor histopathology and metastatic tendency. Multicenter external validation is needed to determine the benefits of using radiomics versus standard PET metrics in clinical practice.

PET segmentation of bulky tumors: Strategies and workflows to improve inter-observer variability

Background

PET-based tumor delineation is an error prone and labor intensive part of image analysis. Especially for patients with advanced disease showing bulky tumor FDG load, segmentations are challenging. Reducing the amount of user-interaction in the segmentation might help to facilitate segmentation tasks especially when labeling bulky and complex tumors. Therefore, this study reports on segmentation workflows/strategies that may reduce the inter-observer variability for large tumors with complex shapes with different levels of user-interaction.

Methods

Twenty PET images of bulky tumors were delineated independently by six observers using four strategies: (I) manual, (II) interactive threshold-based, (III) interactive threshold-based segmentation with the additional presentation of the PET-gradient image and (IV) the selection of the most reasonable result out of four established semi-automatic segmentation algorithms (Select-the-best approach). The segmentations were compared using Jaccard coefficients (JC) and percentage volume differences. To obtain a reference standard, a majority vote (MV) segmentation was calculated including all segmentations of experienced observers. Performed and MV segmentations were compared regarding positive predictive value (PPV), sensitivity (SE), and percentage volume differences.

Results

The results show that with decreasing user-interaction the inter-observer variability decreases. JC values and percentage volume differences of Select-the-best and a workflow including gradient information were significantly better than the measurements of the other segmentation strategies (p-value<0.01). Interactive threshold-based and manual segmentations also result in significant lower and more variable PPV/SE values when compared with the MV segmentation.

Conclusions

FDG PET segmentations of bulky tumors using strategies with lower user-interaction showed less inter-observer variability. None of the methods led to good results in all cases, but use of either the gradient or the Select-the-best workflow did outperform the other strategies tested and may be a good candidate for fast and reliable labeling of bulky and heterogeneous tumors.

Experimental Multicenter and Multivendor Evaluation of the Performance of PET Radiomic Features Using 3-Dimensionally Printed Phantom Inserts

The sensitivity of radiomic features to several confounding factors, such as reconstruction settings, makes clinical use challenging. To investigate the impact of harmonized image reconstructions on feature consistency, a multicenter phantom study was performed using 3-dimensionally printed phantom inserts reflecting realistic tumor shapes and heterogeneity uptakes. Methods: Tumors extracted from real PET/CT scans of patients with non-small cell lung cancer served as model for three 3-dimensionally printed inserts. Different heterogeneity pattern were realized by printing separate compartments that could be filled with different activity solutions. The inserts were placed in the National Electrical Manufacturers Association image-quality phantom and scanned various times. First, a list-mode scan was acquired and 5 statistically equal replicates were reconstructed. Second, the phantom was scanned 4 times on the same scanner. Third, the phantom was scanned on 6 PET/CT systems. All images were reconstructed using EANM Research Ltd. (EARL)-compliant and locally clinically preferred reconstructions. EARL-compliant reconstructions were performed without (EARL1) or with (EARL2) point-spread function. Images were analyzed with and without resampling to 2-mm cubic voxels. Images were discretized with a fixed bin width (FBW) of 0.25 and a fixed bin number (FBN) of 64. The intraclass correlation coefficient (ICC) of each scan setup was calculated and compared across reconstruction settings. An ICC above 0.75 was regarded as high. Results: The percentage of features yielding a high ICC was largest for the statistically equal replicates (70%-91% for FBN; 90%-96% for FBW discretization). For scans acquired on the same system, the percentage decreased, but most features still resulted in a high ICC (FBN, 52%-63%; FBW, 75%-85%). The percentage of features yielding a high ICC decreased more in the multicenter setting. In this case, the percentage of features yielding a high ICC was larger for images reconstructed with EARL-compliant reconstructions: for example, 40% for EARL1 and 60% for EARL2 versus 21% for the clinically preferred setting for FBW discretization. When discretized with FBW and resampled to isotropic voxels, this benefit was more pronounced. Conclusion: EARL-compliant reconstructions harmonize a wide range of radiomic features. FBW discretization and a sampling to isotropic voxels enhances the benefits of EARL-compliant reconstructions.

Multicenter CT phantoms public dataset for radiomics reproducibility tests

PURPOSE

The aim of this paper is to describe a public, open-access, computed tomography (CT) phantom image set acquired at three centers and collected especially for radiomics reproducibility research. The dataset is useful to test radiomic features reproducibility with respect to various parameters, such as acquisition settings, scanners, and reconstruction algorithms.

ACQUISITION AND VALIDATION METHODS

Three phantoms were scanned in three independent institutions. Images of the following phantoms were acquired: Catphan 700 and COPDGene Phantom II (Phantom Laboratory, Greenwich, NY, USA), and the Triple modality 3D Abdominal Phantom (CIRS, Norfolk, VA, USA). Data were collected at three Dutch medical centers: MAASTRO Clinic (Maastricht, NL), Radboud University Medical Center (Nijmegen, NL), and University Medical Center Groningen (Groningen, NL) with scanners from two different manufacturers Siemens Healthcare and Philips Healthcare. The following acquisition parameter were varied in the phantom scans: slice thickness, reconstruction kernels, and tube current.

DATA FORMAT AND USAGE NOTES

We made the dataset publically available on the Dutch instance of "Extensible Neuroimaging Archive Toolkit-XNAT" (https://xnat.bmia.nl). The dataset is freely available and reusable with attribution (Creative Commons 3.0 license).

POTENTIAL APPLICATIONS

Our goal was to provide a findable, open-access, annotated, and reusable CT phantom dataset for radiomics reproducibility studies. Reproducibility testing and harmonization are fundamental requirements for wide generalizability of radiomics-based clinical prediction models. It is highly desirable to include only reproducible features into models, to be more assured of external validity across hitherto unseen contexts. In this view, phantom data from different centers represent a valuable source of information to exclude CT radiomic features that may already be unstable with respect to simplified structures and tightly controlled scan settings. The intended extension of our shared dataset is to include other modalities and phantoms with more realistic lesion simulations.

RaCaT: An open source and easy to use radiomics calculator tool

Purpose

The widely known field ‘Radiomics’ aims to provide an extensive image based phenotyping of e.g. tumors using a wide variety of feature values extracted from medical images. Therefore, it is of utmost importance that feature values calculated by different institutes follow the same feature definitions. For this purpose, the imaging biomarker standardization initiative (IBSI) provides detailed mathematical feature descriptions, as well as (mathematical) test phantoms and corresponding reference feature values. We present here an easy to use radiomic feature calculator, RaCaT, which provides the calculation of a large number of radiomic features for all kind of medical images which are in compliance with the standard.

Methods

The calculator is implemented in C++ and comes as a standalone executable. Therefore, it can be easily integrated in any programming language, but can also be called from the command line. No programming skills are required to use the calculator. The software architecture is highly modularized so that it is easily extendible. The user can also download the source code, adapt it if needed and build the calculator from source. The calculated feature values are compliant with the ones provided by the IBSI standard. Source code, example files for the software configuration, and documentation can be found online on GitHub (https://github.com/ellipfaehlerUMCG/RaCat).

Results

The comparison with the standard values shows that all calculated features as well as image preprocessing steps, comply with the IBSI standard. The performance is also demonstrated on clinical examples.

Conclusions

The authors successfully implemented an easy to use Radiomics calculator that can be called from any programming language or from the command line. Image preprocessing and feature settings and calculations can be adjusted by the user.

Repeatability of 18 F-FDG PET radiomic features: A phantom study to explore sensitivity to image reconstruction settings, noise, and delineation method

Background

¹⁸F‐fluoro‐2‐deoxy‐D‐Glucose positron emission tomography (¹⁸F‐FDG PET) radiomics has the potential to guide the clinical decision making in cancer patients, but validation is required before radiomics can be implemented in the clinical setting. The aim of this study was to explore how feature space reduction and repeatability of ¹⁸F‐FDG PET radiomic features are affected by various sources of variation such as underlying data (e.g., object size and uptake), image reconstruction methods and settings, noise, discretization method, and delineation method.

Methods

The NEMA image quality phantom was scanned with various sphere‐to‐background ratios (SBR), simulating different activity uptakes, including spheres with low uptake, that is, SBR smaller than 1. Furthermore, images of a phantom containing 3D printed inserts reflecting realistic heterogeneity uptake patterns were acquired. Data were reconstructed using various matrix sizes, reconstruction algorithms, and scan durations (noise). For every specific reconstruction and noise level, ten statistically equal replicates were generated. The phantom inserts were delineated using CT and PET‐based segmentation methods. A total of 246 radiomic features was extracted from each image dataset. Images were discretized with a fixed number of 64 bins (FBN) and a fixed bin width (FBW) of 0.25 for the high and a FBW of 0.05 for the low uptake data. In terms of feature reduction, we determined the impact of these factors on the composition of feature clusters, which were defined on the basis of Spearman's correlation matrices. To assess feature repeatability, the intraclass correlation coefficient was calculated over the ten replicates.

Results

In general, larger spheres with high uptake resulted in better repeatability compared to smaller low uptake spheres. In terms of repeatability, features extracted from heterogeneous phantom inserts were comparable to features extracted from bigger high uptake spheres. For example, for an EARL‐compliant reconstruction, larger and smaller high uptake spheres yielded good repeatability for 32% and 30% of the features, while the heterogeneous inserts resulted in 34% repeatable features. For the low uptake spheres, this was the case for 22% and 20% of the features for bigger and smaller spheres, respectively. Images reconstructed with point‐spread‐function (PSF) resulted in the highest repeatability when compared with OSEM or time‐of‐flight, for example, 53%, 30%, and 32% of repeatable features, respectively (for unsmoothed data, discretized with FBN, 300 s scan duration). Reducing image noise (increasing scan duration and smoothing) and using CT‐based segmentation for the low uptake spheres yielded improved repeatability. FBW discretization resulted in higher repeatability than FBN discretization, for example, 89% and 35% of the features, respectively (for the EARL‐compliant reconstruction and larger high uptake spheres).

Conclusion

Feature space reduction and repeatability of ¹⁸F‐FDG PET radiomic features depended on all studied factors. The high sensitivity of PET radiomic features to image quality suggests that a high level of image acquisition and preprocessing standardization is required to be used as clinical imaging biomarker.

SMART (SiMulAtion and ReconsTruction) PET: an efficient PET simulation-reconstruction tool

Background

Positron-emission tomography (PET) simulators are frequently used for development and performance evaluation of segmentation methods or quantitative uptake metrics. To date, most PET simulation tools are based on Monte Carlo simulations, which are computationally demanding. Other analytical simulation tools lack the implementation of time of flight (TOF) or resolution modelling (RM). In this study, a fast and easy-to-use PET simulation-reconstruction package, SiMulAtion and ReconsTruction (SMART)-PET, is developed and validated, which includes both TOF and RM. SMART-PET, its documentation and instructions to calibrate the tool to a specific PET/CT system are available on Zenodo.

SMART-PET allows the fast generation of 3D PET images. As input, it requires one image representing the activity distribution and one representing the corresponding CT image/attenuation map. It allows the user to adjust different parameters, such as reconstruction settings (TOF/RM), noise level or scan duration. Furthermore, a random spatial shift can be included, representing patient repositioning. To evaluate the tool, simulated images were compared with real scan data of the NEMA NU 2 image quality phantom. The scan was acquired as a 60-min list-mode scan and reconstructed with and without TOF and/or RM. For every reconstruction setting, ten statistically equivalent images, representing 30, 60, 120 and 300 s scan duration, were generated. Simulated and real-scan data were compared regarding coefficient of variation in the phantom background and activity recovery coefficients (RCs) of the spheres. Furthermore, standard deviation images of each of the ten statistically equivalent images were compared.

Results

SMART-PET produces images comparable to actual phantom data. The image characteristics of simulated and real PET images varied in similar ways as function of reconstruction protocols and noise levels. The change in image noise with variation of simulated TOF settings followed the theoretically expected behaviour. RC as function of sphere size agreed within 0.3–11% between simulated and actual phantom data.

Conclusions

SMART-PET allows for rapid and easy simulation of PET data. The user can change various acquisition and reconstruction settings (including RM and TOF) and noise levels. The images obtained show similar image characteristics as those seen in actual phantom data.

Electronic supplementary material

The online version of this article (10.1186/s40658-018-0215-x) contains supplementary material, which is available to authorized users.

Impact of PET/CT system, reconstruction protocol, data analysis method, and repositioning on PET/CT precision: An experimental evaluation using an oncology and brain phantom

Purpose

In longitudinal oncological and brain PET/CT studies, it is important to understand the repeatability of quantitative PET metrics in order to assess change in tracer uptake. The present studies were performed in order to assess precision as function of PET/CT system, reconstruction protocol, analysis method, scan duration (or image noise), and repositioning in the field of view.

Methods

Multiple (repeated) scans have been performed using a NEMA image quality (IQ) phantom and a 3D Hoffman brain phantom filled with ¹⁸F solutions on two systems. Studies were performed with and without randomly (< 2 cm) repositioning the phantom and all scans (12 replicates for IQ phantom and 10 replicates for Hoffman brain phantom) were performed at equal count statistics. For the NEMA IQ phantom, we studied the recovery coefficients (RC) of the maximum (SUV_max), peak (SUV_peak), and mean (SUV_mean) uptake in each sphere as a function of experimental conditions (noise level, reconstruction settings, and phantom repositioning). For the 3D Hoffman phantom, the mean activity concentration was determined within several volumes of interest and activity recovery and its precision was studied as function of experimental conditions.

Results

The impact of phantom repositioning on RC precision was mainly seen on the Philips Ingenuity PET/CT, especially in the case of smaller spheres (< 17 mm diameter, P < 0.05). This effect was much smaller for the Siemens Biograph system. When exploring SUV_max, SUV_peak, or SUV_mean of the spheres in the NEMA IQ phantom, it was observed that precision depended on phantom repositioning, reconstruction algorithm, and scan duration, with SUV_max being most and SUV_peak least sensitive to phantom repositioning. For the brain phantom, regional averaged SUVs were only minimally affected by phantom repositioning (< 2 cm).

Conclusion

The precision of quantitative PET metrics depends on the combination of reconstruction protocol, data analysis methods and scan duration (scan statistics). Moreover, precision was also affected by phantom repositioning but its impact depended on the data analysis method in combination with the reconstructed voxel size (tissue fraction effect). This study suggests that for oncological PET studies the use of SUV_peak may be preferred over SUV_max because SUV_peak is less sensitive to patient repositioning/tumor sampling.