The impact of phantom design and material-dependence on repeatability and reproducibility of CT-based radiomics features

Independent validation of CT radiomics models in colorectal liver metastases: predicting local tumour progression after ablation

OBJECTIVES

Independent internal and external validation of three previously published CT-based radiomics models to predict local tumor progression (LTP) after thermal ablation of colorectal liver metastases (CRLM).

MATERIALS AND METHODS

Patients with CRLM treated with thermal ablation were collected from two institutions to collect a new independent internal and external validation cohort. Ablation zones (AZ) were delineated on portal venous phase CT 2-8 weeks post-ablation. Radiomics features were extracted from the AZ and a 10 mm peri-ablational rim (PAR) of liver parenchyma around the AZ. Three previously published prediction models (clinical, radiomics, combined) were tested without retraining. LTP was defined as new tumor foci appearing next to the AZ up to 24 months post-ablation.

RESULTS

The internal cohort included 39 patients with 68 CRLM and the external cohort 52 patients with 78 CRLM. 34/146 CRLM developed LTP after a median follow-up of 24 months (range 5-139). The median time to LTP was 8 months (range 2-22). The combined clinical-radiomics model yielded a c-statistic of 0.47 (95%CI 0.30-0.64) in the internal cohort and 0.50 (95%CI 0.38-0.62) in the external cohort, compared to 0.78 (95%CI 0.65-0.87) in the previously published original cohort. The radiomics model yielded c-statistics of 0.46 (95%CI 0.29-0.63) and 0.39 (95%CI 0.28-0.52), and the clinical model 0.51 (95%CI 0.34-0.68) and 0.51 (95%CI 0.39-0.63) in the internal and external cohort, respectively.

CONCLUSION

The previously published results for prediction of LTP after thermal ablation of CRLM using clinical and radiomics models were not reproducible in independent internal and external validation.

CLINICAL RELEVANCE STATEMENT

Local tumour progression after thermal ablation of CRLM cannot yet be predicted with the use of CT radiomics of the ablation zone and peri-ablational rim. These results underline the importance of validation of radiomics results to test for reproducibility in independent cohorts.

KEY POINTS

• Previous research suggests CT radiomics models have the potential to predict local tumour progression after thermal ablation in colorectal liver metastases, but independent validation is lacking. • In internal and external validation, the previously published models were not able to predict local tumour progression after ablation. • Radiomics prediction models should be investigated in independent validation cohorts to check for reproducibility.

CT Cadaveric dataset for Radiomics features stability assessment in lumbar vertebrae

Radiomics features (RFs) studies have showed limitations in the reproducibility of RFs in different acquisition settings. To date, reproducibility studies using CT images mainly rely on phantoms, due to the harness of patient exposure to X-rays. The provided CadAIver dataset has the aims of evaluating how CT scanner parameters effect radiomics features on cadaveric donor. The dataset comprises 112 unique CT acquisitions of a cadaveric truck acquired on 3 different CT scanners varying KV, mA, field-of-view, and reconstruction kernel settings. Technical validation of the CadAIver dataset comprises a comprehensive univariate and multivariate GLM approach to assess stability of each RFs extracted from lumbar vertebrae. The complete dataset is publicly available to be applied for future research in the RFs field, and could foster the creation of a collaborative open CT image database to increase the sample size, the range of available scanners, and the available body districts.

Predictive value of multiple imaging predictive models for spread through air spaces of lung adenocarcinoma: A systematic review and network meta‑analysis

Spread Through Air Spaces (STAS) is involved in lung adenocarcinoma (LUAD) recurrence, where cancer cells spread into adjacent lung tissue, impacting surgical planning and prognosis assessment. Radiomics-based models show promise in predicting STAS preoperatively, enhancing surgical precision and prognostic evaluations. The present study performed network meta-analysis to assess the predictive efficacy of imaging models for STAS in LUAD. Data were systematically sourced from PubMed, Embase, Scopus, Wiley and Web of Science, according to the Cochrane Handbook for Systematic Reviews of Interventions) and A Measurement Tool to Assess systematic Reviews 2. Using Stata software v17.0 for meta-analysis, surface under the cumulative ranking area (SUCRA) was applied to identify the most effective diagnostic method. Quality assessments were performed using Cochrane Collaboration's risk-of-bias tool and publication bias was assessed using Deeks' funnel plot. The analysis encompassed 14 articles, involving 3,734 patients, and assessed 17 predictive models for STAS in LUAD. According to comprehensive analysis of SUCRA, the machine learning (ML)_Peri_tumour model had the highest accuracy (56.5), the Features_computed tomography (CT) model had the highest sensitivity (51.9) and the positron emission tomography (pet)_CT model had the highest specificity (53.9). ML_Peri_tumour model had the highest predictive performance. The accuracy was as follows: ML_Peri_tumour vs. Features_CT [relative risk (RR)=1.14; 95% confidence interval (CI), 0.99-1.32]; ML_Peri_tumour vs. ML_Tumour (RR=1.04; 95% CI, 0.83-1.30) and ML_Peri_tumour vs. pet_CT (RR=1.04; 95% CI, 0.84-1.29). Comparative analyses revealed heightened predictive accuracy of the ML_Peri_tumour compared with other models. Nonetheless, the field of radiological feature analysis for STAS prediction remains nascent, necessitating improvements in technical reproducibility and comprehensive model evaluation.

Impacts of Adaptive Statistical Iterative Reconstruction-V and Deep Learning Image Reconstruction Algorithms on Robustness of CT Radiomics Features: Opportunity for Minimizing Radiomics Variability Among Scans of Different Dose Levels

This study aims to investigate the influence of adaptive statistical iterative reconstruction-V (ASIR-V) and deep learning image reconstruction (DLIR) on CT radiomics feature robustness. A standardized phantom was scanned under single-energy CT (SECT) and dual-energy CT (DECT) modes at standard and low (20 and 10 mGy) dose levels. Images of SECT 120 kVp and corresponding DECT 120 kVp-like virtual monochromatic images were generated with filtered back-projection (FBP), ASIR-V at 40% (AV-40) and 100% (AV-100) blending levels, and DLIR algorithm at low (DLIR-L), medium (DLIR-M), and high (DLIR-H) strength levels. Ninety-four features were extracted via Pyradiomics. Reproducibility of features was calculated between standard and low dose levels, between reconstruction algorithms in reference to FBP images, and within scan mode, using intraclass correlation coefficient (ICC) and concordance correlation coefficient (CCC). The average percentage of features with ICC > 0.90 and CCC > 0.90 between the two dose levels was 21.28% and 20.75% in AV-40 images, and 39.90% and 35.11% in AV-100 images, respectively, and increased from 15.43 to 45.22% and from 15.43 to 44.15% with an increasing strength level of DLIR. The average percentage of features with ICC > 0.90 and CCC > 0.90 in reference to FBP images was 26.07% and 25.80% in AV-40 images, and 18.88% and 18.62% in AV-100 images, respectively, and decreased from 27.93 to 17.82% and from 27.66 to 17.29% with an increasing strength level of DLIR. DLIR and ASIR-V algorithms showed low reproducibility in reference to FBP images, while the high-strength DLIR algorithm provides an opportunity for minimizing radiomics variability due to dose reduction.

A novel CT-responsive hydrogel for the construction of an organ simulation phantom for the repeatability and stability study of radiomic features

Radiomic features have demonstrated reliable outcomes in tumor grading and detecting precancerous lesions in medical imaging analysis. However, the repeatability and stability of these features have faced criticism. In this study, we aim to enhance the repeatability and stability of radiomic features by introducing a novel CT-responsive hydrogel material. The newly developed CT-responsive hydrogel, mineralized by in situ metal ions, exhibits exceptional repeatability, stability, and uniformity. Moreover, by adjusting the concentration of metal ions, it achieves remarkable CT similarity comparable to that of human organs on CT scans. To create a phantom, the hydrogel was molded into a universal model, displaying controllable CT values ranging from 53 HU to 58 HU, akin to human liver tissue. Subsequently, 1218 radiomic features were extracted from the CT-responsive hydrogel organ simulation phantom. Impressively, 85-97.2% of the extracted features exhibited good repeatability and stability during coefficient of variability analysis. This finding emphasizes the potential of CT-responsive hydrogel in consistently extracting the same features, providing a novel approach to address the issue of repeatability in radiomic features.

Identification of reproducible radiomic features from on-board volumetric images: A multi-institutional phantom study

BACKGROUND

Radiomics analysis using on-board volumetric images has attracted research attention as a method for predicting prognosis during treatment; however, the lack of standardization is still one of the main concerns.

PURPOSE

This study investigated the factors that influence the reproducibility of radiomic features extracted from on-board volumetric images using an anthropomorphic radiomics phantom. Furthermore, a phantom experiment was conducted with different treatment machines from multiple institutions as external validation to identify reproducible radiomic features.

METHODS

The phantom was designed to be 35 × 20 × 20 cm with eight types of heterogeneous spheres (⌀ = 1, 2, and 3 cm). On-board volumetric images were acquired using 15 treatment machines from eight institutions. Of these, kilovoltage cone-beam computed tomography (kV-CBCT) image data acquired from four treatment machines at one institution were used as an internal evaluation dataset to explore the reproducibility of radiomic features. The remaining image data, including kV-CBCT, megavoltage-CBCT (MV-CBCT), and megavoltage computed tomography (MV-CT) provided by seven different institutions (11 treatment machines), were used as an external validation dataset. A total of 1,302 radiomic features, including 18 first-order, 75 texture, 465 (i.e., 93 × 5) Laplacian of Gaussian (LoG) filter-based, and 744 (i.e., 93 × 8) wavelet filter-based features, were extracted within the spheres. The intraclass correlation coefficient (ICC) was calculated to explore feature repeatability and reproducibility using an internal evaluation dataset. Subsequently, the coefficient of variation (COV) was calculated to validate the feature variability of external institutions. An absolute ICC exceeding 0.85 or COV under 5% was considered indicative of a highly reproducible feature.

RESULTS

For internal evaluation, ICC analysis showed that the median percentage of radiomic features with high repeatability was 95.2%. The ICC analysis indicated that the median percentages of highly reproducible features for inter-tube current, reconstruction algorithm, and treatment machine were decreased by 20.8%, 29.2%, and 33.3%, respectively. For external validation, the COV analysis showed that the median percentage of reproducible features was 31.5%. A total of 16 features, including nine LoG filter-based and seven wavelet filter-based features, were indicated as highly reproducible features. The gray-level run-length matrix (GLRLM) was classified as containing the most frequent features (N = 8), followed by the gray-level dependence matrix (N = 7) and gray-level co-occurrence matrix (N = 1) features.

CONCLUSIONS

We developed the standard phantom for radiomics analysis of kV-CBCT, MV-CBCT, and MV-CT images. With this phantom, we revealed that the differences in the treatment machine and image reconstruction algorithm reduce the reproducibility of radiomic features from on-board volumetric images. Specifically, the most reproducible features for external validation were LoG or wavelet filter-based GLRLM features. However, the acceptability of the identified features should be examined in advance at each institution before applying the findings to prognosis prediction.

Artificial intelligence-driven radiomics study in cancer: the role of feature engineering and modeling

Modern medicine is reliant on various medical imaging technologies for non-invasively observing patients' anatomy. However, the interpretation of medical images can be highly subjective and dependent on the expertise of clinicians. Moreover, some potentially useful quantitative information in medical images, especially that which is not visible to the naked eye, is often ignored during clinical practice. In contrast, radiomics performs high-throughput feature extraction from medical images, which enables quantitative analysis of medical images and prediction of various clinical endpoints. Studies have reported that radiomics exhibits promising performance in diagnosis and predicting treatment responses and prognosis, demonstrating its potential to be a non-invasive auxiliary tool for personalized medicine. However, radiomics remains in a developmental phase as numerous technical challenges have yet to be solved, especially in feature engineering and statistical modeling. In this review, we introduce the current utility of radiomics by summarizing research on its application in the diagnosis, prognosis, and prediction of treatment responses in patients with cancer. We focus on machine learning approaches, for feature extraction and selection during feature engineering and for imbalanced datasets and multi-modality fusion during statistical modeling. Furthermore, we introduce the stability, reproducibility, and interpretability of features, and the generalizability and interpretability of models. Finally, we offer possible solutions to current challenges in radiomics research.

Robustness of radiomics features of virtual unenhanced and virtual monoenergetic images in dual-energy CT among different imaging platforms and potential role of CT number variability

OBJECTIVES

To evaluate robustness of dual-energy CT (DECT) radiomics features of virtual unenhanced (VUE) image and virtual monoenergetic image (VMI) among different imaging platforms.

METHODS

A phantom with sixteen clinical-relevant densities was scanned on ten DECT platforms with comparable scan parameters. Ninety-four radiomic features were extracted via Pyradiomics from VUE images and VMIs at energy level of 70 keV (VMI_70keV). Test-retest repeatability was assessed by Bland-Altman analysis. Inter-platform reproducibility of VUE images and VMI_70keV was evaluated by coefficient of variation (CV) and quartile coefficient of dispersion (QCD) among platforms, and by intraclass correlation coefficient (ICC) and concordance correlation coefficient (CCC) between platform pairs. The correlation between variability of CT number radiomics reproducibility was estimated.

RESULTS

92.02% and 92.87% of features were repeatable between scan-rescans for VUE images and VMI_70keV, respectively. Among platforms, 11.30% and 28.39% features of VUE images, and 15.16% and 28.99% features of VMI_70keV were with CV < 10% and QCD < 10%. The average percentages of radiomics features with ICC > 0.90 and CCC > 0.90 between platform pairs were 10.00% and 9.86% in VUE images and 11.23% and 11.23% in VMI_70keV. The CT number inter-platform reproducibility using CV and QCD showed negative correlations with percentage of the first-order radiomics features with CV < 10% and QCD < 10%, in both VUE images and VMI_70keV (r² 0.3870-0.6178, all p < 0.001).

CONCLUSIONS

The majority of DECT radiomics features were non-reproducible. The differences in CT number were considered as an indicator of inter-platform DECT radiomics variation. Critical relevance statement: The majority of radiomics features extracted from the VUE images and the VMI70keV were non-reproducible among platforms, while synchronizing energy levels of VMI to reduce the CT number value variability may be a potential way to mitigate radiomics instability.

Deep learning image reconstruction algorithm reduces image noise while alters radiomics features in dual-energy CT in comparison with conventional iterative reconstruction algorithms: a phantom study

OBJECTIVES

To compare image quality between a deep learning image reconstruction (DLIR) algorithm and conventional iterative reconstruction (IR) algorithms in dual-energy CT (DECT) and to assess the impact of these algorithms on radiomics robustness.

METHODS

A phantom with clinical-relevant densities was imaged on seven DECT scanners with the same voxel size using typical abdominal-pelvis examination protocols. On one DECT scanner, raw data were reconstructed using both conventional IR (adaptive statistical iterative reconstruction-V, ASIR-V) and DLIR. Nine sets of corresponding images were generated on other six DECT scanners using scanner-equipped conventional IR. Regions of interest were delineated through rigid registrations. Image quality was compared. Pyradiomics platform was used for radiomics feature extraction. Test-retest repeatability was assessed by Bland-Altman analysis for repeated scans. Inter-reconstruction algorithm reproducibility between conventional IR and DLIR was tested by intraclass correlation coefficient (ICC) and concordance correlation coefficient (CCC). Inter-scanner reproducibility was evaluated by coefficient of variation (CV) and quartile coefficient of dispersion (QCD). Robust features were identified.

RESULTS

DLIR significantly improved image quality. Ninety-four radiomics features were extracted and nine features were considered as robust. 93.87% features were repeatable between repeated scans. ASIR-V images showed higher reproducibility to other conventional IR than DLIR (ICC mean, 0.603 vs 0.558, p = 0.001; CCC mean, 0.554 vs 0.510, p = 0.004). 7.45% and 26.83% features were reproducible among scanners evaluated by CV and QCD, respectively.

CONCLUSIONS

DLIR improves quality of DECT images but may alter radiomics features compared to conventional IR. Nine robust DECT radiomics features were identified.

KEY POINTS

• DLIR improves DECT image quality in terms of signal-to-noise ratio and contrast-to-noise ratio compared with ASIR-V and showed the highest noise reduction rate and lowest peak frequency shift. • Most of radiomics features are repeatable between repeated DECT scans, while inter-reconstruction algorithm reproducibility between conventional IR and DLIR, and inter-scanner reproducibility, are low. • Although DLIR may alter radiomics features compared to IR algorithms, nine radiomics features survived repeatability and reproducibility analysis among DECT scanners and reconstruction algorithms, which allows further validation and clinical-relevant analysis.

Radiomic and quantitative-semantic models of low-dose computed tomography for predicting the poorly differentiated invasive non-mucinous pulmonary adenocarcinoma

PURPOSE

Poorly differentiated invasive non-mucinous pulmonary adenocarcinoma (IPA), based on the novel grading system, was related to poor prognosis, with a high risk of lymph node metastasis and local recurrence. This study aimed to build the radiomic and quantitative-semantic models of low-dose computed tomography (LDCT) to preoperatively predict the poorly differentiated IPA in nodules with solid component, and compare their diagnostic performance with radiologists.

MATERIALS AND METHODS

A total of 396 nodules from 388 eligible patients, who underwent LDCT scan within 2 weeks before surgery and were pathologically diagnosed with IPA, were retrospectively enrolled between July 2018 and December 2021. Nodules were divided into two independent cohorts according to scanners: primary cohort (195 well/moderate differentiated and 64 poorly differentiated) and validation cohort (104 well/moderate differentiated and 33 poorly differentiated). The radiomic and quantitative-semantic models were built using multivariable logistic regression. The diagnostic performance of the models and radiologists was assessed by area under curve (AUC) of receiver operating characteristic (ROC) curve, accuracy, sensitivity, and specificity.

RESULTS

No significant differences of AUCs were found between the radiomic and quantitative-semantic model in primary and validation cohorts (0.921 vs. 0.923, P = 0.846 and 0.938 vs. 0.911, P = 0.161). Both the models outperformed three radiologists in the validation cohort (all P < 0.05).

CONCLUSIONS

The radiomic and quantitative-semantic models of LDCT, which could identify the poorly differentiated IPA with excellent diagnostic performance, might provide guidance for therapeutic decision making, such as choosing appropriate surgical method or adjuvant chemotherapy.

Radiomics: Status quo and future challenges

Noninvasive imaging (computed tomography, magnetic resonance imaging, endoscopic ultrasonography, and positron emission tomography) as an important part of the clinical workflow in the clinic, but it still provides limited information for diagnosis, treatment effect evaluation and prognosis prediction. In addition, judgment and diagnoses made by experts are usually based on multiple years of experience and subjective impression which lead to variable results in the same case. With accumulation of medical imaging data, radiomics emerges as a relatively new approach for analysis. Via artificial intelligence techniques, high-throughput quantitative data which is invisible to the naked eyes extracted from original images can be used in the process of patients’ management. Several studies have evaluated radiomics combined with clinical factors, pathological, or genetic information would assist in the diagnosis, particularly in the prediction of biological characteristics, risk of recurrence, and survival with encouraging results. In various clinical settings, there are limitations and challenges needing to be overcome before transformation. Therefore, we summarize the concepts and method of radiomics including image acquisition, region of interest segmentation, feature extraction and model development. We also set forth the current applications of radiomics in clinical routine. At last, the limitations and related deficiencies of radiomics are pointed out to direct the future opportunities and development.

Radiomics in clinical trials: perspectives on standardization

The term biomarker is used to describe a biological measure of the disease behavior. The existing imaging biomarkers are associated with the known tissue biological characteristics and follow a well-established roadmap to be implemented in routine clinical practice. Recently, a new quantitative imaging analysis approach named radiomics has emerged. It refers to the extraction of a large number of advanced imaging features with high-throughput computing. Extensive research has demonstrated its value in predicting disease behavior, progression, and response to therapeutic options. However, there are numerous challenges to establishing it as a clinically viable solution, including lack of reproducibility and transparency. The data-driven nature also does not offer insights into the underpinning biology of the observed relationships. As such, additional effort is needed to establish it as a qualified biomarker to inform clinical decisions. Here we review the technical difficulties encountered in the clinical applications of radiomics and current effort in addressing some of these challenges in clinical trial designs. By addressing these challenges, the true potential of radiomics can be unleashed.