Evaluation of the dependence of radiomic features on the machine learning model

Breast MRI radiomics and machine learning-based predictions of response to neoadjuvant chemotherapy - How are they affected by variations in tumor delineation?

Manual delineation of volumes of interest (VOIs) by experts is considered the gold-standard method in radiomics analysis. However, it suffers from inter- and intra-operator variability. A quantitative assessment of the impact of variations in these delineations on the performance of the radiomics predictors is required to develop robust radiomics based prediction models. In this study, we developed radiomics models for the prediction of pathological complete response to neoadjuvant chemotherapy in patients with two different breast cancer subtypes based on contrast-enhanced magnetic resonance imaging acquired prior to treatment (baseline MRI scans). Different mathematical operations such as erosion, smoothing, dilation, randomization, and ellipse fitting were applied to the original VOIs delineated by experts to simulate variations of segmentation masks. The effects of such VOI modifications on various steps of the radiomics workflow, including feature extraction, feature selection, and prediction performance, were evaluated. Using manual tumor VOIs and radiomics features extracted from baseline MRI scans, an AUC of up to 0.96 and 0.89 was achieved for human epidermal growth receptor 2 positive and triple-negative breast cancer, respectively. For smoothing and erosion, VOIs yielded the highest number of robust features and the best prediction performance, while ellipse fitting and dilation lead to the lowest robustness and prediction performance for both breast cancer subtypes. At most 28% of the selected features were similar to manual VOIs when different VOI delineation data were used. Differences in VOI delineation affect different steps of radiomics analysis, and their quantification is therefore important for development of standardized radiomics research.

Explainable Precision Medicine in Breast MRI: A Combined Radiomics and Deep Learning Approach for the Classification of Contrast Agent Uptake

In DCE-MRI, the degree of contrast uptake in normal fibroglandular tissue, i.e., background parenchymal enhancement (BPE), is a crucial biomarker linked to breast cancer risk and treatment outcome. In accordance with the Breast Imaging Reporting & Data System (BI-RADS), it should be visually classified into four classes. The susceptibility of such an assessment to inter-reader variability highlights the urgent need for a standardized classification algorithm. In this retrospective study, the first post-contrast subtraction images for 27 healthy female subjects were included. The BPE was classified slice-wise by two expert radiologists. The extraction of radiomic features from segmented BPE was followed by dataset splitting and dimensionality reduction. The latent representations were then utilized as inputs to a deep neural network classifying BPE into BI-RADS classes. The network's predictions were elucidated at the radiomic feature level with Shapley values. The deep neural network achieved a BPE classification accuracy of 84 ± 2% (p-value < 0.00001). Most of the misclassifications involved adjacent classes. Different radiomic features were decisive for the prediction of each BPE class underlying the complexity of the decision boundaries. A highly precise and explainable pipeline for BPE classification was achieved without user- or algorithm-dependent radiomic feature selection.

Utilization of texture features of volumetric ADC maps in differentiating between serous cystadenoma and intraductal papillary neoplasms

INTRODUCTION

The rising incidence of incidental detection of pancreatic cystic neoplasms has compelled radiologists to determine new diagnostic methods for the differentiation of various kinds of lesions. We aim to demonstrate the utility of texture features extracted from ADC maps in differentiating intraductal papillary mucinous neoplasms (IPMN) from serous cystadenomas (SCA).

METHODS

This retrospective study was performed on 136 patients (IPMN = 87, SCA = 49) split into testing and training datasets. A total of 851 radiomics features were extracted from volumetric contours drawn by an expert radiologist on ADC maps of the lesions. LASSO regression analysis was used to determine the most predictive set of features and a radiomics score was developed based on their respective coefficients. A hyper-optimized support vector machine was then utilized to classify the lesions based on their radiomics score.

RESULTS

A total of four Wavelet features (LHL/GLCM/LCM2, HLL/GLCM/LCM2, /LLL/First Order/90percent, /LLL/GLCM/MCC) were selected from all of the features to be included in our classifier. The classifier was optimized by altering hyperparameters and the trained model was applied to the validation dataset. The model achieved a sensitivity of 92.8, specificity of 90%, and an AUC of 0.97 in the training data set, and a sensitivity of 83.3%, specificity of 66.7%, and AUC of 0.90 in the testing dataset.

CONCLUSION

A support vector machine model trained and validated on volumetric texture features extracted from ADC maps showed the possible beneficence of these features in differentiating IPMNs from SCAs. These results are in line with previous regarding the role of ADC maps in classifying cystic lesions and offers new evidence regarding the role of texture features in differentiation of potentially neoplastic and benign lesions.

The effect of data resampling methods in radiomics

Radiomic datasets can be class-imbalanced, for instance, when the prevalence of diseases varies notably, meaning that the number of positive samples is much smaller than that of negative samples. In these cases, the majority class may dominate the model's training and thus negatively affect the model's predictive performance, leading to bias. Therefore, resampling methods are often utilized to class-balance the data. However, several resampling methods exist, and neither their relative predictive performance nor their impact on feature selection has been systematically analyzed. In this study, we aimed to measure the impact of nine resampling methods on radiomic models utilizing a set of fifteen publicly available datasets regarding their predictive performance. Furthermore, we evaluated the agreement and similarity of the set of selected features. Our results show that applying resampling methods did not improve the predictive performance on average. On specific datasets, slight improvements in predictive performance (+ 0.015 in AUC) could be seen. A considerable disagreement on the set of selected features was seen (only 28.7% of features agreed), which strongly impedes feature interpretability. However, selected features are similar when considering their correlation (82.9% of features correlated on average).

The effect of feature normalization methods in radiomics

OBJECTIVES

In radiomics, different feature normalization methods, such as z-Score or Min-Max, are currently utilized, but their specific impact on the model is unclear. We aimed to measure their effect on the predictive performance and the feature selection.

METHODS

We employed fifteen publicly available radiomics datasets to compare seven normalization methods. Using four feature selection and classifier methods, we used cross-validation to measure the area under the curve (AUC) of the resulting models, the agreement of selected features, and the model calibration. In addition, we assessed whether normalization before cross-validation introduces bias.

RESULTS

On average, the difference between the normalization methods was relatively small, with a gain of at most + 0.012 in AUC when comparing the z-Score (mean AUC: 0.707 ± 0.102) to no normalization (mean AUC: 0.719 ± 0.107). However, on some datasets, the difference reached + 0.051. The z-Score performed best, while the tanh transformation showed the worst performance and even decreased the overall predictive performance. While quantile transformation performed, on average, slightly worse than the z-Score, it outperformed all other methods on one out of three datasets. The agreement between the features selected by different normalization methods was only mild, reaching at most 62%. Applying the normalization before cross-validation did not introduce significant bias.

CONCLUSION

The choice of the feature normalization method influenced the predictive performance but depended strongly on the dataset. It strongly impacted the set of selected features.

CRITICAL RELEVANCE STATEMENT

Feature normalization plays a crucial role in the preprocessing and influences the predictive performance and the selected features, complicating feature interpretation.

KEY POINTS

• The impact of feature normalization methods on radiomic models was measured. • Normalization methods performed similarly on average, but differed more strongly on some datasets. • Different methods led to different sets of selected features, impeding feature interpretation. • Model calibration was not largely affected by the normalization method.

Radiomic texture analysis based on neurite orientation dispersion and density imaging to differentiate glioblastoma from solitary brain metastasis

BACKGROUND

We created discriminative models of different regions of interest (ROIs) using radiomic texture features of neurite orientation dispersion and density imaging (NODDI) and evaluated the feasibility of each model in differentiating glioblastoma multiforme (GBM) from solitary brain metastasis (SBM).

METHODS

We conducted a retrospective study of 204 patients with GBM (n = 146) or SBM (n = 58). Radiomic texture features were extracted from five ROIs based on three metric maps (intracellular volume fraction, orientation dispersion index, and isotropic volume fraction of NODDI), including necrosis, solid tumors, peritumoral edema, tumor bulk volume (TBV), and abnormal bulk volume. Four feature selection methods and eight classifiers were used for the radiomic texture feature selection and model construction. Receiver operating characteristic (ROC) curve analysis was used to evaluate the diagnostic performance of the models. Routine magnetic resonance imaging (MRI) radiomic texture feature models generated in the same manner were used for the horizontal comparison.

RESULTS

NODDI-radiomic texture analysis based on TBV subregions exhibited the highest accuracy (although nonsignificant) in differentiating GBM from SBM, with area under the ROC curve (AUC) values of 0.918 and 0.882 in the training and test datasets, respectively, compared to necrosis (AUCtraining:0.845, AUCtest:0.714), solid tumor (AUCtraining:0.852, AUCtest:0.821), peritumoral edema (AUCtraining:0.817, AUCtest:0.762), and ABV (AUCtraining:0.834, AUCtest:0.779). The performance of the five ROI radiomic texture models in routine MRI was inferior to that of the NODDI-radiomic texture model.

CONCLUSION

Preoperative NODDI-radiomic texture analysis based on TBV subregions shows great potential for distinguishing GBM from SBM.

Universal multi-factor feature selection method for radiomics-based brain tumor classification

Brain tumor mortality is high, and accurate classification before treatment can improve patient prognosis. Radiomics, which extracts numerous features from medical images, has been widely applied in brain tumor classification studies. Feature selection (FS) is a critical step in radiomics because it reduces redundant information and enhances classification performance. However, the lack of universal FS methods limits the development of radiomics-based brain tumor classification studies. To address this issue, we summarize the characteristics of the FS methods used in related studies and propose a universal method based on three selection factors called triple-factor cascaded selection (TFCS). Particularly, these factors correspond to the correlation between features and task labels, interdependence among features, and role of features in the model. The TFCS method divides FS into two steps. First, it utilizes mutual information to select features that are strongly correlated with the task and contain less redundant information. Recursive feature elimination is then employed to obtain the subset with the best classification performance. To validate the universality of the TFCS, we conducted experiments on seven datasets containing 13 brain tumor classification tasks and evaluated the overall performance using five types of indicators. Results: TFCS exhibited excellent overall performance for all tasks. Compared to the 13 related methods, it takes less time, has moderate parsimony, the best classification performance, adaptability, and stability, and shows better universality. Our study demonstrates that the reasonable utilization of multiple factors can enhance FS performance and provide new insights for future method design.

Radiomic analysis for early differentiation of lung cancer recurrence from fibrosis in patients treated with lung stereotactic ablative radiotherapy

Objective. The development of radiation-induced fibrosis after stereotactic ablative radiotherapy (SABR) can obscure follow-up images and delay detection of a local recurrence in early-stage lung cancer patients. The objective of this study was to develop a radiomics model for computer-assisted detection of local recurrence and fibrosis for an earlier timepoint (<1 year) after the SABR treatment.Approach. This retrospective clinical study included CT images (n= 107) of 66 patients treated with SABR. A z-score normalization technique was used for radiomic feature standardization across scanner protocols. The training set for the radiomics model consisted of CT images (66 patients; 22 recurrences and 44 fibrosis) obtained at 24 months (median) follow-up. The test set included CT-images of 41 patients acquired at 5-12 months follow-up. Combinations of four widely used machine learning techniques (support vector machines, gradient boosting, random forests (RF), and logistic regression) and feature selection methods (Relief feature scoring, maximum relevance minimum redundancy, mutual information maximization, forward feature selection, and LASSO) were investigated. Pyradiomics was used to extract 106 radiomic features from the CT-images for feature selection and classification.Main results. An RF + LASSO model scored the highest in terms of AUC (0.87) and obtained a sensitivity of 75% and a specificity of 88% in identifying a local recurrence in the test set. In the training set, 86% accuracy was achieved using five-fold cross-validation. Delong's test indicated that AUC achieved by the RF+LASSO is significantly better than 11 other machine learning models presented here. The top three radiomic features: interquartile range (first order), Cluster Prominence (GLCM), and Autocorrelation (GLCM), were revealed as differentiating a recurrence from fibrosis with this model.Significance. The radiomics model selected, out of multiple machine learning and feature selection algorithms, was able to differentiate a recurrence from fibrosis in earlier follow-up CT-images with a high specificity rate and satisfactory sensitivity performance.

A systematic review of radiomics in giant cell tumor of bone (GCTB): the potential of analysis on individual radiomics feature for identifying genuine promising imaging biomarkers

PURPOSE

To systematically assess the quality of radiomics research in giant cell tumor of bone (GCTB) and to test the feasibility of analysis at the level of radiomics feature.

METHODS

We searched PubMed, Embase, Web of Science, China National Knowledge Infrastructure, and Wanfang Data to identify articles of GCTB radiomics until 31 July 2022. The studies were assessed by radiomics quality score (RQS), transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD) statement, checklist for artificial intelligence in medical imaging (CLAIM), and modified quality assessment of diagnostic accuracy studies (QUADAS-2) tool. The radiomic features selected for model development were documented.

RESULTS

Nine articles were included. The average of the ideal percentage of RQS, the TRIPOD adherence rate and the CLAIM adherence rate were 26%, 56%, and 57%, respectively. The risk of bias and applicability concerns were mainly related to the index test. The shortness in external validation and open science were repeatedly emphasized. In GCTB radiomics models, the gray level co-occurrence matrix features (40%), first order features (28%), and gray-level run-length matrix features (18%) were most selected features out of all reported features. However, none of the individual feature has appeared repeatably in multiple studies. It is not possible to meta-analyze radiomics features at present.

CONCLUSION

The quality of GCTB radiomics studies is suboptimal. The reporting of individual radiomics feature data is encouraged. The analysis at the level of radiomics feature has potential to generate more practicable evidence for translating radiomics into clinical application.

Multi-centre radiomics for prediction of recurrence following radical radiotherapy for head and neck cancers: Consequences of feature selection, machine learning classifiers and batch-effect harmonization

Background and purpose

Radiomics models trained with limited single institution data are often not reproducible and generalisable. We developed radiomics models that predict loco-regional recurrence within two years of radiotherapy with private and public datasets and their combinations, to simulate small and multi-institutional studies and study the responsiveness of the models to feature selection, machine learning algorithms, centre-effect harmonization and increased dataset sizes.

Materials and methods

562 patients histologically confirmed and treated for locally advanced head-and-neck cancer (LA-HNC) from two public and two private datasets; one private dataset exclusively reserved for validation. Clinical contours of primary tumours were not recontoured and were used for Pyradiomics based feature extraction. ComBat harmonization was applied, and LASSO-Logistic Regression (LR) and Support Vector Machine (SVM) models were built. 95% confidence interval (CI) of 1000 bootstrapped area-under-the-Receiver-operating-curves (AUC) provided predictive performance. Responsiveness of the models' performance to the choice of feature selection methods, ComBat harmonization, machine learning classifier, single and pooled data was evaluated.

Results

LASSO and SelectKBest selected 14 and 16 features, respectively; three were overlapping. Without ComBat, the LR and SVM models for three institutional data showed AUCs (CI) of 0.513 (0.481-0.559) and 0.632 (0.586-0.665), respectively. Performances following ComBat revealed AUCs of 0.559 (0.536-0.590) and 0.662 (0.606-0.690), respectively. Compared to single cohort AUCs (0.562-0.629), SVM models from pooled data performed significantly better at AUC = 0.680.

Conclusions

Multi-institutional retrospective data accentuates the existing variabilities that affect radiomics. Carefully designed prospective, multi-institutional studies and data sharing are necessary for clinically relevant head-and-neck cancer prognostication models.

Are deep models in radiomics performing better than generic models? A systematic review

BACKGROUND

Application of radiomics proceeds by extracting and analysing imaging features based on generic morphological, textural, and statistical features defined by formulas. Recently, deep learning methods were applied. It is unclear whether deep models (DMs) can outperform generic models (GMs).

METHODS

We identified publications on PubMed and Embase to determine differences between DMs and GMs in terms of receiver operating area under the curve (AUC).

RESULTS

Of 1,229 records (between 2017 and 2021), 69 studies were included, 61 (88%) on tumours, 68 (99%) retrospective, and 39 (56%) single centre; 30 (43%) used an internal validation cohort; and 18 (26%) applied cross-validation. Studies with independent internal cohort had a median training sample of 196 (range 41-1,455); those with cross-validation had only 133 (43-1,426). Median size of validation cohorts was 73 (18-535) for internal and 94 (18-388) for external. Considering the internal validation, in 74% (49/66), the DMs performed better than the GMs, vice versa in 20% (13/66); no difference in 6% (4/66); and median difference in AUC 0.045. On the external validation, DMs were better in 65% (13/20), GMs in 20% (4/20) cases; no difference in 3 (15%); and median difference in AUC 0.025. On internal validation, fused models outperformed GMs and DMs in 72% (20/28), while they were worse in 14% (4/28) and equal in 14% (4/28); median gain in AUC was + 0.02. On external validation, fused model performed better in 63% (5/8), worse in 25% (2/8), and equal in 13% (1/8); median gain in AUC was + 0.025.

CONCLUSIONS

Overall, DMs outperformed GMs but in 26% of the studies, DMs did not outperform GMs.