Measuring the bias of incorrect application of feature selection when using cross-validation in radiomics

Leveraging radiomics and machine learning to differentiate radiation necrosis from recurrence in patients with brain metastases

OBJECTIVE

Radiation necrosis (RN) can be difficult to radiographically discern from tumor progression after stereotactic radiosurgery (SRS). The objective of this study was to investigate the utility of radiomics and machine learning (ML) to differentiate RN from recurrence in patients with brain metastases treated with SRS.

METHODS

Patients with brain metastases treated with SRS who developed either RN or tumor reccurence were retrospectively identified. Image preprocessing and radiomic feature extraction were performed using ANTsPy and PyRadiomics, yielding 105 features from MRI T1-weighted post-contrast (T1c), T2, and fluid-attenuated inversion recovery (FLAIR) images. Univariate analysis assessed significance of individual features. Multivariable analysis employed various classifiers on features identified as most discriminative through feature selection. ML models were evaluated through cross-validation, selecting the best model based on area under the receiver operating characteristic (ROC) curve (AUC). Specificity, sensitivity, and F1 score were computed.

RESULTS

Sixty-six lesions from 55 patients were identified. On univariate analysis, 27 features from the T1c sequence were statistically significant, while no features were significant from the T2 or FLAIR sequences. For clinical variables, only immunotherapy use after SRS was significant. Multivariable analysis of features from the T1c sequence yielded an AUC of 76.2% (standard deviation [SD] ± 12.7%), with specificity and sensitivity of 75.5% (± 13.4%) and 62.3% (± 19.6%) in differentiating radionecrosis from recurrence.

CONCLUSIONS

Radiomics with ML may assist the diagnostic ability of distinguishing RN from tumor recurrence after SRS. Further work is needed to validate this in a larger multi-institutional cohort and prospectively evaluate it's utility in patient care.

Applying oversampling before cross-validation will lead to high bias in radiomics

Class imbalance is often unavoidable for radiomic data collected from clinical routine. It can create problems during classifier training since the majority class could dominate the minority class. Consequently, resampling methods like oversampling or undersampling are applied to the data to class-balance the data. However, the resampling must not be applied upfront to all data because it would lead to data leakage and, therefore, to erroneous results. This study aims to measure the extent of this bias. Five-fold cross-validation with 30 repeats was performed using a set of 15 radiomic datasets to train predictive models. The training involved two scenarios: first, the models were trained correctly by applying the resampling methods during the cross-validation. Second, the models were trained incorrectly by performing the resampling on all the data before cross-validation. The bias was defined empirically as the difference between the best-performing models in both scenarios in terms of area under the receiver operating characteristic curve (AUC), sensitivity, specificity, balanced accuracy, and the Brier score. In addition, a simulation study was performed on a randomly generated dataset for verification. The results demonstrated that incorrectly applying the oversampling methods to all data resulted in a large positive bias (up to 0.34 in AUC, 0.33 in sensitivity, 0.31 in specificity, and 0.37 in balanced accuracy). The bias depended on the data balance, and approximately an increase of 0.10 in the AUC was observed for each increase in imbalance. The models also showed a bias in calibration measured using the Brier score, which differed by up to -0.18 between the correctly and incorrectly trained models. The undersampling methods were not affected significantly by bias. These results emphasize that any resampling method should be applied correctly only to the training data to avoid data leakage and, subsequently, biased model performance and calibration.

Can multi-modal radiomics using pretreatment ultrasound and tomosynthesis predict response to neoadjuvant systemic treatment in breast cancer?

OBJECTIVES

Response assessment to neoadjuvant systemic treatment (NAST) to guide individualized treatment in breast cancer is a clinical research priority. We aimed to develop an intelligent algorithm using multi-modal pretreatment ultrasound and tomosynthesis radiomics features in addition to clinical variables to predict pathologic complete response (pCR) prior to the initiation of therapy.

METHODS

We used retrospective data on patients who underwent ultrasound and tomosynthesis before starting NAST. We developed a support vector machine algorithm using pretreatment ultrasound and tomosynthesis radiomics features in addition to patient and tumor variables to predict pCR status (ypT0 and ypN0). Findings were compared to the histopathologic evaluation of the surgical specimen. The main outcome measures were area under the curve (AUC) and false-negative rate (FNR).

RESULTS

We included 720 patients, 504 in the development set and 216 in the validation set. Median age was 51.6 years and 33.6% (242 of 720) achieved pCR. The addition of radiomics features significantly improved the performance of the algorithm (AUC 0.72 to 0.81; p = 0.007). The FNR of the multi-modal radiomics and clinical algorithm was 6.7% (10 of 150 with missed residual cancer). Surface/volume ratio at tomosynthesis and peritumoral entropy characteristics at ultrasound were the most relevant radiomics. Hormonal receptors and HER-2 status were the most important clinical predictors.

CONCLUSION

A multi-modal machine learning algorithm with pretreatment clinical, ultrasound, and tomosynthesis radiomics features may aid in predicting residual cancer after NAST. Pending prospective validation, this may facilitate individually tailored NAST regimens.

CLINICAL RELEVANCE STATEMENT

Multi-modal radiomics using pretreatment ultrasound and tomosynthesis showed significant improvement in assessing response to NAST compared to an algorithm using clinical variables only. Further prospective validation of our findings seems warranted to enable individualized predictions of NAST outcomes.

KEY POINTS

• We proposed a multi-modal machine learning algorithm with pretreatment clinical, ultrasound, and tomosynthesis radiomics features to predict response to neoadjuvant breast cancer treatment. • Compared with the clinical algorithm, the AUC of this integrative algorithm is significantly higher. • Used prior to the initiative of therapy, our algorithm can identify patients who will experience pathologic complete response following neoadjuvant therapy with a high negative predictive value.

Ultrasound Radiomics Features to Identify Patients With Triple-Negative Breast Cancer: A Retrospective, Single-Center Study

OBJECTIVES

Patients with triple-negative breast cancer (TNBC) exhibit a fast tumor growth rate and poor survival outcomes. In this study, we aimed to develop and compare intelligent algorithms using ultrasound radiomics features in addition to clinical variables to identify patients with TNBC prior to histopathologic diagnosis.

METHODS

We used single-center, retrospective data of patients who underwent ultrasound before histopathologic verification and subsequent neoadjuvant systemic treatment (NAST). We developed a logistic regression with an elastic net penalty algorithm using pretreatment ultrasound radiomics features in addition to patient and tumor variables to identify patients with TNBC. Findings were compared to the histopathologic evaluation of the biopsy specimen. The main outcome measure was the area under the curve (AUC).

RESULTS

We included 1161 patients, 813 in the development set and 348 in the validation set. Median age was 50.1 years and 24.4% (283 of 1161) had TNBC. The integrative model using radiomics and clinical information showed significantly better performance in identifying TNBC compared to the radiomics model (AUC: 0.71, 95% confidence interval [CI]: 0.65-0.76 versus 0.64, 95% CI: 0.57-0.71, P = .004). The five most important variables were cN status, shape surface volume ratio (SA:V), gray level co-occurrence matrix (GLCM) correlation, gray level dependence matrix (GLDM) dependence nonuniformity normalized, and age. Patients with TNBC were more often categorized as BI-RADS 4 than BI-RADS 5 compared to non-TNBC patients (P = .002).

CONCLUSION

A machine learning algorithm showed promising potential to identify patients with TNBC using ultrasound radiomics features and clinical information prior to histopathologic evaluation.

Support vector machine prediction of individual Autism Diagnostic Observation Schedule (ADOS) scores based on neural responses during live eye-to-eye contact

Social difficulties during interactions with others are central to autism spectrum disorder (ASD). Understanding the links between these social difficulties and their underlying neural processes is a primary aim focused on improved diagnosis and treatment. In keeping with this goal, we have developed a multivariate classification method based on neural data acquired by functional near infrared spectroscopy, fNIRS, during live eye-to-eye contact with adults who were either typically developed (TD) or individuals with ASD. The ASD diagnosis was based on the gold-standard Autism Diagnostic Observation Schedule (ADOS) which also provides an index of symptom severity. Using a nested cross-validation method, a support vector machine (SVM) was trained to discriminate between ASD and TD groups based on the neural responses during eye-to-eye contact. ADOS scores were not applied in the classification training. To test the hypothesis that SVM identifies neural activity patterns related to one of the neural mechanisms underlying the behavioral symptoms of ASD, we determined the correlation coefficient between the SVM scores and the individual ADOS scores. Consistent with the hypothesis, the correlation between observed and predicted ADOS scores was 0.72 (p < 0.002). Findings suggest that multivariate classification methods combined with the live interaction paradigm of eye-to-eye contact provide a promising approach to link neural processes and social difficulties in individuals with ASD.

Classification performance bias between training and test sets in a limited mammography dataset

OBJECTIVES

To assess the performance bias caused by sampling data into training and test sets in a mammography radiomics study.

METHODS

Mammograms from 700 women were used to study upstaging of ductal carcinoma in situ. The dataset was repeatedly shuffled and split into training (n = 400) and test cases (n = 300) forty times. For each split, cross-validation was used for training, followed by an assessment of the test set. Logistic regression with regularization and support vector machine were used as the machine learning classifiers. For each split and classifier type, multiple models were created based on radiomics and/or clinical features.

RESULTS

Area under the curve (AUC) performances varied considerably across the different data splits (e.g., radiomics regression model: train 0.58-0.70, test 0.59-0.73). Performances for regression models showed a tradeoff where better training led to worse testing and vice versa. Cross-validation over all cases reduced this variability, but required samples of 500+ cases to yield representative estimates of performance.

CONCLUSIONS

In medical imaging, clinical datasets are often limited to relatively small size. Models built from different training sets may not be representative of the whole dataset. Depending on the selected data split and model, performance bias could lead to inappropriate conclusions that might influence the clinical significance of the findings.

ADVANCES IN KNOWLEDGE

Performance bias can result from model testing when using limited datasets. Optimal strategies for test set selection should be developed to ensure study conclusions are appropriate.

The effect of data resampling methods in radiomics

Radiomic datasets can be class-imbalanced, for instance, when the prevalence of diseases varies notably, meaning that the number of positive samples is much smaller than that of negative samples. In these cases, the majority class may dominate the model's training and thus negatively affect the model's predictive performance, leading to bias. Therefore, resampling methods are often utilized to class-balance the data. However, several resampling methods exist, and neither their relative predictive performance nor their impact on feature selection has been systematically analyzed. In this study, we aimed to measure the impact of nine resampling methods on radiomic models utilizing a set of fifteen publicly available datasets regarding their predictive performance. Furthermore, we evaluated the agreement and similarity of the set of selected features. Our results show that applying resampling methods did not improve the predictive performance on average. On specific datasets, slight improvements in predictive performance (+ 0.015 in AUC) could be seen. A considerable disagreement on the set of selected features was seen (only 28.7% of features agreed), which strongly impedes feature interpretability. However, selected features are similar when considering their correlation (82.9% of features correlated on average).

Identification of integrated proteomics and transcriptomics signature of alcohol-associated liver disease using machine learning

Distinguishing between alcohol-associated hepatitis (AH) and alcohol-associated cirrhosis (AC) remains a diagnostic challenge. In this study, we used machine learning with transcriptomics and proteomics data from liver tissue and peripheral mononuclear blood cells (PBMCs) to classify patients with alcohol-associated liver disease. The conditions in the study were AH, AC, and healthy controls. We processed 98 PBMC RNAseq samples, 55 PBMC proteomic samples, 48 liver RNAseq samples, and 53 liver proteomic samples. First, we built separate classification and feature selection pipelines for transcriptomics and proteomics data. The liver tissue models were validated in independent liver tissue datasets. Next, we built integrated gene and protein expression models that allowed us to identify combined gene-protein biomarker panels. For liver tissue, we attained 90% nested-cross validation accuracy in our dataset and 82% accuracy in the independent validation dataset using transcriptomic data. We attained 100% nested-cross validation accuracy in our dataset and 61% accuracy in the independent validation dataset using proteomic data. For PBMCs, we attained 83% and 89% accuracy with transcriptomic and proteomic data, respectively. The integration of the two data types resulted in improved classification accuracy for PBMCs, but not liver tissue. We also identified the following gene-protein matches within the gene-protein biomarker panels: CLEC4M-CLC4M, GSTA1-GSTA2 for liver tissue and SELENBP1-SBP1 for PBMCs. In this study, machine learning models had high classification accuracy for both transcriptomics and proteomics data, across liver tissue and PBMCs. The integration of transcriptomics and proteomics into a multi-omics model yielded improvement in classification accuracy for the PBMC data. The set of integrated gene-protein biomarkers for PBMCs show promise toward developing a liquid biopsy for alcohol-associated liver disease.

The effect of feature normalization methods in radiomics

OBJECTIVES

In radiomics, different feature normalization methods, such as z-Score or Min-Max, are currently utilized, but their specific impact on the model is unclear. We aimed to measure their effect on the predictive performance and the feature selection.

METHODS

We employed fifteen publicly available radiomics datasets to compare seven normalization methods. Using four feature selection and classifier methods, we used cross-validation to measure the area under the curve (AUC) of the resulting models, the agreement of selected features, and the model calibration. In addition, we assessed whether normalization before cross-validation introduces bias.

RESULTS

On average, the difference between the normalization methods was relatively small, with a gain of at most + 0.012 in AUC when comparing the z-Score (mean AUC: 0.707 ± 0.102) to no normalization (mean AUC: 0.719 ± 0.107). However, on some datasets, the difference reached + 0.051. The z-Score performed best, while the tanh transformation showed the worst performance and even decreased the overall predictive performance. While quantile transformation performed, on average, slightly worse than the z-Score, it outperformed all other methods on one out of three datasets. The agreement between the features selected by different normalization methods was only mild, reaching at most 62%. Applying the normalization before cross-validation did not introduce significant bias.

CONCLUSION

The choice of the feature normalization method influenced the predictive performance but depended strongly on the dataset. It strongly impacted the set of selected features.

CRITICAL RELEVANCE STATEMENT

Feature normalization plays a crucial role in the preprocessing and influences the predictive performance and the selected features, complicating feature interpretation.

KEY POINTS

• The impact of feature normalization methods on radiomic models was measured. • Normalization methods performed similarly on average, but differed more strongly on some datasets. • Different methods led to different sets of selected features, impeding feature interpretation. • Model calibration was not largely affected by the normalization method.

Early diabetic eye damage: Comparing detection methods using diagnostic power

It is now clear that retinal neuropathy precedes classical microvascular retinopathy in diabetes. Therefore, tests that underpin useful new endpoints must provide high diagnostic power well before the onset of moderate diabetic retinopathy. Hence, we compare detection methods of early diabetic eye damage. We reviewed data from a range of functional and structural studies of early diabetic eye disease and computed standardized effect size as a measure of diagnostic power, allowing the studies to be compared quantitatively. We then derived minimum performance criteria for tests to provide useful clinical endpoints. This included the criteria that tests should be rapid and easy so that children with type 1 diabetes can be followed into adulthood with the same tests. We also defined attributes that lend test data to further improve performance using Machine/Deep Learning. Data from a new form of objective perimetry suggested that the criteria are achievable.

Development of a prediction model for head and neck volume reduction by clinical factors, dose-volume histogram parameters and radiomics in head and neck cancer†

In external radiotherapy of head and neck (HN) cancers, the reduction of irradiation accuracy due to HN volume reduction often causes a problem. Adaptive radiotherapy (ART) can effectively solve this problem; however, its application to all cases is impractical because of cost and time. Therefore, finding priority cases is essential. This study aimed to predict patients with HN cancers are more likely to need ART based on a quantitative measure of large HN volume reduction and evaluate model accuracy. The study included 172 cases of patients with HN cancer who received external irradiation. The HN volume was calculated using cone-beam computed tomography (CT) for irradiation-guided radiotherapy for all treatment fractions and classified into two groups: cases with a large reduction in the HN volume and cases without a large reduction. Radiomic features were extracted from the primary gross tumor volume (GTV) and nodal GTV of the planning CT. To develop the prediction model, four feature selection methods and two machine-learning algorithms were tested. Predictive performance was evaluated by the area under the curve (AUC), accuracy, sensitivity and specificity. Predictive performance was the highest for the random forest, with an AUC of 0.662. Furthermore, its accuracy, sensitivity and specificity were 0.692, 0.700 and 0.813, respectively. Selected features included radiomic features of the primary GTV, human papillomavirus in oropharyngeal cancer and the implementation of chemotherapy; thus, these features might be related to HN volume change. Our model suggested the potential to predict ART requirements based on HN volume reduction .

Radiomics: The New Promise for Differentiating Progression, Recurrence, Pseudoprogression, and Radionecrosis in Glioma and Glioblastoma Multiforme

Glioma and glioblastoma multiform (GBM) remain among the most debilitating and life-threatening brain tumors. Despite advances in diagnosing approaches, patient follow-up after treatment (surgery and chemoradiation) is still challenging for differentiation between tumor progression/recurrence, pseudoprogression, and radionecrosis. Radiomics emerges as a promising tool in initial diagnosis, grading, and survival prediction in patients with glioma and can help differentiate these post-treatment scenarios. Preliminary published studies are promising about the role of radiomics in post-treatment glioma/GBM. However, this field faces significant challenges, including a lack of evidence-based solid data, scattering publication, heterogeneity of studies, and small sample sizes. The present review explores radiomics's capabilities in following patients with glioma/GBM status post-treatment and to differentiate tumor progression, recurrence, pseudoprogression, and radionecrosis.

Spatial assessments in texture analysis: what the radiologist needs to know

To date, studies investigating radiomics-based predictive models have tended to err on the side of data-driven or exploratory analysis of many thousands of extracted features. In particular, spatial assessments of texture have proven to be especially adept at assessing for features of intratumoral heterogeneity in oncologic imaging, which likewise may correspond with tumor biology and behavior. These spatial assessments can be generally classified as spatial filters, which detect areas of rapid change within the grayscale in order to enhance edges and/or textures within an image, or neighborhood-based methods, which quantify gray-level differences of neighboring pixels/voxels within a set distance. Given the high dimensionality of radiomics datasets, data dimensionality reduction methods have been proposed in an attempt to optimize model performance in machine learning studies; however, it should be noted that these approaches should only be applied to training data in order to avoid information leakage and model overfitting. While area under the curve of the receiver operating characteristic is perhaps the most commonly reported assessment of model performance, it is prone to overestimation when output classifications are unbalanced. In such cases, confusion matrices may be additionally reported, whereby diagnostic cut points for model predicted probability may hold more clinical significance to clinical colleagues with respect to related forms of diagnostic testing.

A CT-Based Clinical, Radiological and Radiomic Machine Learning Model for Predicting Malignancy of Solid Renal Tumors (UroCCR-75)

BACKGROUND

Differentiating benign from malignant renal tumors is important for patient management, and it may be improved by quantitative CT features analysis including radiomic.

PURPOSE

This study aimed to compare performances of machine learning models using bio-clinical, conventional radiologic and 3D-radiomic features for the differentiation of benign and malignant solid renal tumors using pre-operative multiphasic contrast-enhanced CT examinations.

MATERIALS AND METHODS

A unicentric retrospective analysis of prospectively acquired data from a national kidney cancer database was conducted between January 2016 and December 2020. Histologic findings were obtained by robotic-assisted partial nephrectomy. Lesion images were semi-automatically segmented, allowing for a 3D-radiomic features extraction in the nephrographic phase. Conventional radiologic parameters such as shape, content and enhancement were combined in the analysis. Biological and clinical features were obtained from the national database. Eight machine learning (ML) models were trained and validated using a ten-fold cross-validation. Predictive performances were evaluated comparing sensitivity, specificity, accuracy and AUC.

RESULTS

A total of 122 patients with 132 renal lesions, including 111 renal cell carcinomas (RCCs) (111/132, 84%) and 21 benign tumors (21/132, 16%), were evaluated (58 +/- 14 years, men 74%). Unilaterality (100/111, 90% vs. 13/21, 62%; p = 0.02), necrosis (81/111, 73% vs. 8/21, 38%; p = 0.02), lower values of tumor/cortex ratio at portal time (0.61 vs. 0.74, p = 0.01) and higher variation of tumor/cortex ratio between arterial and portal times (0.22 vs. 0.05, p = 0.008) were associated with malignancy. A total of 35 radiomics features were selected, and "intensity mean value" was associated with RCCs in multivariate analysis (OR = 0.99). After ten-fold cross-validation, a C5.0Tree model was retained for its predictive performances, yielding a sensitivity of 95%, specificity of 42%, accuracy of 87% and AUC of 0.74.

CONCLUSION

Our machine learning-based model combining clinical, radiologic and radiomics features from multiphasic contrast-enhanced CT scans may help differentiate benign from malignant solid renal tumors.

Are deep models in radiomics performing better than generic models? A systematic review

BACKGROUND

Application of radiomics proceeds by extracting and analysing imaging features based on generic morphological, textural, and statistical features defined by formulas. Recently, deep learning methods were applied. It is unclear whether deep models (DMs) can outperform generic models (GMs).

METHODS

We identified publications on PubMed and Embase to determine differences between DMs and GMs in terms of receiver operating area under the curve (AUC).

RESULTS

Of 1,229 records (between 2017 and 2021), 69 studies were included, 61 (88%) on tumours, 68 (99%) retrospective, and 39 (56%) single centre; 30 (43%) used an internal validation cohort; and 18 (26%) applied cross-validation. Studies with independent internal cohort had a median training sample of 196 (range 41-1,455); those with cross-validation had only 133 (43-1,426). Median size of validation cohorts was 73 (18-535) for internal and 94 (18-388) for external. Considering the internal validation, in 74% (49/66), the DMs performed better than the GMs, vice versa in 20% (13/66); no difference in 6% (4/66); and median difference in AUC 0.045. On the external validation, DMs were better in 65% (13/20), GMs in 20% (4/20) cases; no difference in 3 (15%); and median difference in AUC 0.025. On internal validation, fused models outperformed GMs and DMs in 72% (20/28), while they were worse in 14% (4/28) and equal in 14% (4/28); median gain in AUC was + 0.02. On external validation, fused model performed better in 63% (5/8), worse in 25% (2/8), and equal in 13% (1/8); median gain in AUC was + 0.025.

CONCLUSIONS

Overall, DMs outperformed GMs but in 26% of the studies, DMs did not outperform GMs.

Machine learning-based CT radiomics model to discriminate the primary and secondary intracranial hemorrhage

It is challenging to distinguish between primary and secondary intracranial hemorrhage (ICH) purely by imaging data, and the two forms of ICHs are treated differently. This study aims to evaluate the potential of CT-based machine learning to identify the etiology of ICHs and compare the effectiveness of two regions of interest (ROI) sketching methods. A total of 1702 radiomic features were extracted from the CT brain images of 238 patients with acute ICH. We used the Select K Best method, least absolute shrinkage, and selection operator logistic regression to select the most discriminable features with a support vector machine to build a classifier model. Then, a ten-fold cross-validation strategy was employed to evaluate the performance of the classifier. From all quantitative CT-based imaging features obtained by two sketch methods, eighteen features were selected respectively. The radiomics model outperformed radiologists in distinguishing between primary and secondary ICH in both the volume of interest and the three-layer ROI sketches. As a result, a machine learning-based CT radiomics model can improve the accuracy of identifying primary and secondary ICH. A three-layer ROI sketch can identify primary versus secondary ICH based on the CT radiomics method.

Introduction to radiomics for a clinical audience

Radiomics is a rapidly developing field of research focused on the extraction of quantitative features from medical images, thus converting these digital images into minable, high-dimensional data, which offer unique biological information that can enhance our understanding of disease processes and provide clinical decision support. To date, most radiomics research has been focused on oncological applications; however, it is increasingly being used in a raft of other diseases. This review gives an overview of radiomics for a clinical audience, including the radiomics pipeline and the common pitfalls associated with each stage. Key studies in oncology are presented with a focus on both those that use radiomics analysis alone and those that integrate its use with other multimodal data streams. Importantly, clinical applications outside oncology are also presented. Finally, we conclude by offering a vision for radiomics research in the future, including how it might impact our practice as radiologists.

Predicting Soft Tissue Sarcoma Response to Neoadjuvant Chemotherapy Using an MRI-Based Delta-Radiomics Approach

OBJECTIVES

To evaluate the performance of machine learning-augmented MRI-based radiomics models for predicting response to neoadjuvant chemotherapy (NAC) in soft tissue sarcomas.

METHODS

Forty-four subjects were identified retrospectively from patients who received NAC at our institution for pathologically proven soft tissue sarcomas. Only subjects who had both a baseline MRI prior to initiating chemotherapy and a post-treatment scan at least 2 months after initiating chemotherapy and prior to surgical resection were included. 3D ROIs were used to delineate whole-tumor volumes on pre- and post-treatment scans, from which 1708 radiomics features were extracted. Delta-radiomics features were calculated by subtraction of baseline from post-treatment values and used to distinguish treatment response through univariate analyses as well as machine learning-augmented radiomics analyses.

RESULTS

Though only 4.74% of variables overall reached significance at p ≤ 0.05 in univariate analyses, Laws Texture Energy (LTE)-derived metrics represented 46.04% of all such features reaching statistical significance. ROC analyses similarly failed to predict NAC response, with AUCs of 0.40 (95% CI 0.22-0.58) and 0.44 (95% CI 0.26-0.62) for RF and AdaBoost, respectively.

CONCLUSION

Overall, while our result was not able to separate NAC responders from non-responders, our analyses did identify a subset of LTE-derived metrics that show promise for further investigations. Future studies will likely benefit from larger sample size constructions so as to avoid the need for data filtering and feature selection techniques, which have the potential to significantly bias the machine learning procedures.

MAIC-10 brief quality checklist for publications using artificial intelligence and medical images

The use of artificial intelligence (AI) with medical images to solve clinical problems is becoming increasingly common, and the development of new AI solutions is leading to more studies and publications using this computational technology. As a novel research area, the use of common standards to aid AI developers and reviewers as quality control criteria will improve the peer review process. Although some guidelines do exist, their heterogeneity and extension advocate that more explicit and simple schemes should be applied on the publication practice. Based on a review of existing AI guidelines, a proposal which collects, unifies, and simplifies the most relevant criteria was developed. The MAIC-10 (Must AI Criteria-10) checklist with 10 items was implemented as a guide to design studies and evaluate publications related to AI in the field of medical imaging. Articles published in Insights into Imaging in 2021 were selected to calculate their corresponding MAIC-10 quality score. The mean score was found to be 5.6 ± 1.6, with critical items present in most articles, such as "Clinical need", "Data annotation", "Robustness", and "Transparency" present in more than 80% of papers, while improvements in other areas were identified. MAIC-10 was also observed to achieve the highest intra-observer reproducibility when compared to other existing checklists, with an overall reduction in terms of checklist length and complexity. In summary, MAIC-10 represents a short and simple quality assessment tool which is objective, robust and widely applicable to AI studies in medical imaging.

Evaluation of automatic discrimination between benign and malignant prostate tissue in the era of high precision digital pathology

BACKGROUND

Prostate cancer is a major health concern in aging men. Paralleling an aging society, prostate cancer prevalence increases emphasizing the need for efficient diagnostic algorithms.

METHODS

Retrospectively, 106 prostate tissue samples from 48 patients (mean age, [Formula: see text] years) were included in the study. Patients suffered from prostate cancer (n = 38) or benign prostatic hyperplasia (n = 10) and were treated with radical prostatectomy or Holmium laser enucleation of the prostate, respectively. We constructed tissue microarrays (TMAs) comprising representative malignant (n = 38) and benign (n = 68) tissue cores. TMAs were processed to histological slides, stained, digitized and assessed for the applicability of machine learning strategies and open-source tools in diagnosis of prostate cancer. We applied the software QuPath to extract features for shape, stain intensity, and texture of TMA cores for three stainings, H&E, ERG, and PIN-4. Three machine learning algorithms, neural network (NN), support vector machines (SVM), and random forest (RF), were trained and cross-validated with 100 Monte Carlo random splits into 70% training set and 30% test set. We determined AUC values for single color channels, with and without optimization of hyperparameters by exhaustive grid search. We applied recursive feature elimination to feature sets of multiple color transforms.

RESULTS

Mean AUC was above 0.80. PIN-4 stainings yielded higher AUC than H&E and ERG. For PIN-4 with the color transform saturation, NN, RF, and SVM revealed AUC of [Formula: see text], [Formula: see text], and [Formula: see text], respectively. Optimization of hyperparameters improved the AUC only slightly by 0.01. For H&E, feature selection resulted in no increase of AUC but to an increase of 0.02-0.06 for ERG and PIN-4.

CONCLUSIONS

Automated pipelines may be able to discriminate with high accuracy between malignant and benign tissue. We found PIN-4 staining best suited for classification. Further bioinformatic analysis of larger data sets would be crucial to evaluate the reliability of automated classification methods for clinical practice and to evaluate potential discrimination of aggressiveness of cancer to pave the way to automatic precision medicine.

Predictive performance of radiomic models based on features extracted from pretrained deep networks

OBJECTIVES

In radiomics, generic texture and morphological features are often used for modeling. Recently, features extracted from pretrained deep networks have been used as an alternative. However, extracting deep features involves several decisions, and it is unclear how these affect the resulting models. Therefore, in this study, we considered the influence of such choices on the predictive performance.

METHODS

On ten publicly available radiomic datasets, models were trained using feature sets that differed in terms of the utilized network architecture, the layer of feature extraction, the used set of slices, the use of segmentation, and the aggregation method. The influence of these choices on the predictive performance was measured using a linear mixed model. In addition, models with generic features were trained and compared in terms of predictive performance and correlation.

RESULTS

No single choice consistently led to the best-performing models. In the mixed model, the choice of architecture (AUC + 0.016; p < 0.001), the level of feature extraction (AUC + 0.016; p < 0.001), and using all slices (AUC + 0.023; p < 0.001) were highly significant; using the segmentation had a lower influence (AUC + 0.011; p = 0.023), while the aggregation method was insignificant (p = 0.774). Models based on deep features were not significantly better than those based on generic features (p > 0.05 on all datasets). Deep feature sets correlated moderately with each other (r = 0.4), in contrast to generic feature sets (r = 0.89).

CONCLUSIONS

Different choices have a significant effect on the predictive performance of the resulting models; however, for the highest performance, these choices should be optimized during cross-validation.

Feasibility of using CT radiomic signatures for predicting CD8-T cell infiltration and PD-L1 expression in renal cell carcinoma

Objectives

To identify computed tomography (CT)-based radiomic signatures of cluster of differentiation 8 (CD8)-T cell infiltration and programmed cell death ligand 1 (PD-L1) expression levels in patients with clear-cell renal cell carcinoma (ccRCC).

Methods

Seventy-eight patients with pathologically confirmed localized ccRCC, preoperative multiphase CT and tumor resection specimens were enrolled in this retrospective study. Regions of interest (ROI) of the ccRCC volume were manually segmented from the CT images and processed using a radiomics panel comprising of 1708 metrics. The extracted metrics were used as inputs to three machine learning classifiers: Random Forest, AdaBoost, and ElasticNet to create radiomic signatures for CD8-T cell infiltration and PD-L1 expression, respectively.

Results

Using a cut-off of 80 lymphocytes per high power field, 59 % were classified to CD8 highly infiltrated tumors and 41 % were CD8 non highly infiltrated tumors, respectively. An ElasticNet classifier discriminated between these two groups of CD8-T cells with an AUC of 0.68 (95 % CI, 0.55-0.80). In addition, based on tumor proportion score with a cut-off of > 1 % tumor cells expressing PD-L1, 76 % were PD-L1 positive and 24 % were PD-L1 negative. An Adaboost classifier discriminated between PD-L1 positive and PD-L1 negative tumors with an AUC of 0.8 95 % CI: (0.66, 0.95). 3D radiomics metrics of graylevel co-occurrence matrix (GLCM) and graylevel run-length matrix (GLRLM) metrics drove the performance for CD8-Tcell and PD-L1 classification, respectively.

Conclusions

CT-radiomic signatures can differentiate tumors with high CD8-T cell infiltration with moderate accuracy and positive PD-L1 expression with good accuracy in ccRCC.

Radiomic assessment of oesophageal adenocarcinoma: a critical review of 18F-FDG PET/CT, PET/MRI and CT

Objectives

Radiomic models present an avenue to improve oesophageal adenocarcinoma assessment through quantitative medical image analysis. However, model selection is complicated by the abundance of available predictors and the uncertainty of their relevance and reproducibility. This analysis reviews recent research to facilitate precedent-based model selection for prospective validation studies.

Methods

This analysis reviews research on 18F-FDG PET/CT, PET/MRI and CT radiomics in oesophageal adenocarcinoma between 2016 and 2021. Model design, testing and reporting are evaluated according to the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) score and Radiomics Quality Score (RQS). Key results and limitations are analysed to identify opportunities for future research in the area.

Results

Radiomic models of stage and therapeutic response demonstrated discriminative capacity, though clinical applications require greater sensitivity. Although radiomic models predict survival within institutions, generalisability is limited. Few radiomic features have been recommended independently by multiple studies.

Conclusions

Future research must prioritise prospective validation of previously proposed models to further clinical translation.

Supplementary Information

The online version contains supplementary material available at 10.1186/s13244-022-01245-0.

Radiomics-based machine learning models to distinguish between metastatic and healthy bone using lesion-center-based geometric regions of interest

Radiomics-based machine learning classifiers have shown potential for detecting bone metastases (BM) and for evaluating BM response to radiotherapy (RT). However, current radiomics models require large datasets of images with expert-segmented 3D regions of interest (ROIs). Full ROI segmentation is time consuming and oncologists often outline just RT treatment fields in clinical practice. This presents a challenge for real-world radiomics research. As such, a method that simplifies BM identification but does not compromise the power of radiomics is needed. The objective of this study was to investigate the feasibility of radiomics models for BM detection using lesion-center-based geometric ROIs. The planning-CT images of 170 patients with non-metastatic lung cancer and 189 patients with spinal BM were used. The point locations of 631 BM and 674 healthy bone (HB) regions were identified by experts. ROIs with various geometric shapes were centered and automatically delineated on the identified locations, and 107 radiomics features were extracted. Various feature selection methods and machine learning classifiers were evaluated. Our point-based radiomics pipeline was successful in differentiating BM from HB. Lesion-center-based segmentation approach greatly simplifies the process of preparing images for use in radiomics studies and avoids the bottleneck of full ROI segmentation.

Research on Classification of Open-Pit Mineral Exploiting Information Based on OOB RFE Feature Optimization

Mineral exploiting information is an important indicator to reflect regional mineral activities. Accurate extraction of this information is essential to mineral management and environmental protection. In recent years, there are an increasingly large number of pieces of research on land surface information classification by conducting multi-source remote sensing data. However, in order to achieve the best classification result, how to select the optimal feature combination is the key issue. This study creatively combines Out of Bag data with Recursive Feature Elimination (OOB RFE) to optimize the feature combination of the mineral exploiting information of non-metallic building materials in Fujian province, China. We acquired and integrated Ziyuan-1-02D (ZY-1-02D) hyperspectral imagery, landsat-8 multispectral imagery, and Sentinel-1 Synthetic Aperture Radar (SAR) imagery to gain spectrum, heat, polarization, and texture features; also, two machine learning methods were adopted to classify the mineral exploiting information in our study area. After assessment and comparison on accuracy, it proves that the classification generated from our new OOB RFE method, which combine with random forest (RF), can achieve the highest overall accuracy 93.64% (with a kappa coefficient of 0.926). Comparing with Recursive Feature Elimination (RFE) alone, OOB REF can precisely filter the feature combination and lead to optimal result. Under the same feature scheme, RF is effective on classifying the mineral exploiting information of the research field. The feature optimization method and optimal feature combination proposed in our study can provide technical support and theoretical reference for extraction and classification of mineral exploiting information applied in other regions.