Radiomics machine learning study with a small sample size: Single random training-test set split may lead to unreliable results

Towards discovery and implementation of neurophysiologic biomarkers of Alzheimer's disease using entropy methods

Alzheimer's disease (AD) is a prevalent and debilitating neurodegenerative disease that leads to substantial loss of quality of life. Therapies currently available for AD do not modify the disease course and have limited efficacy in symptom control. As such, novel and precise therapies tailored to individual patients' neurophysiologic profiles are needed. Functional neuroimaging tools have demonstrated substantial potential to provide quantifiable insight into brain function in various neurologic disorders, particularly AD. Entropy, a novel analysis for better understanding the nonlinear nature of neurophysiological data, has demonstrated consistent accuracy in disease detection. This literature review characterizes the use of entropy-based analyses from functional neuroimaging tools, including electroencephalography (EEG) and magnetoencephalography (MEG), in patients with AD for disease detection, therapeutic response measurement, and providing clinical insights.

Using machine learning models to estimate Escherichia coli concentration in an irrigation pond from water quality and drone-based RGB imagery data

The rapid and efficient quantification of Escherichia coli concentrations is crucial for monitoring water quality. Remote sensing techniques and machine learning algorithms have been used to detect E. coli in water and estimate its concentrations. The application of these approaches, however, is challenged by limited sample availability and unbalanced water quality datasets. In this study, we estimated the E. coli concentration in an irrigation pond in Maryland, USA, during the summer season using demosaiced natural color (red, green, and blue: RGB) imagery in the visible and infrared spectral ranges, and a set of 14 water quality parameters. We did this by deploying four machine learning models - Random Forest (RF), Gradient Boosting Machine (GBM), Extreme Gradient Boosting (XGB), and K-nearest Neighbor (KNN) - under three data utilization scenarios: water quality parameters only, combined water quality and small unmanned aircraft system (sUAS)-based RGB data, and RGB data only. To select the training and test datasets, we applied two data-splitting methods: ordinary and quantile data splitting. These methods provided a constant splitting ratio in each decile of the E. coli concentration distribution. Quantile data splitting resulted in better model performance metrics and smaller differences between the metrics for both the training and testing datasets. When trained with quantile data splitting after hyperparameter optimization, models RF, GBM, and XGB had R2 values above 0.847 for the training dataset and above 0.689 for the test dataset. The combination of water quality and RGB imagery data resulted in a higher R2 value (>0.896) for the test dataset. Shapley additive explanations (SHAP) of the relative importance of variables revealed that the visible blue spectrum intensity and water temperature were the most influential parameters in the RF model. Demosaiced RGB imagery served as a useful predictor of E. coli concentration in the studied irrigation pond.

Radiomics features outperform standard radiological measurements in detecting femoroacetabular impingement on three-dimensional magnetic resonance imaging

Femoroacetabular impingement (FAI) is a cause of hip pain and can lead to hip osteoarthritis. Radiological measurements obtained from radiographs or magnetic resonance imaging (MRI) are normally used for FAI diagnosis, but they require time-consuming manual interaction, which limits accuracy and reproducibility. This study compares standard radiologic measurements against radiomics features automatically extracted from MRI for the identification of FAI patients versus healthy subjects. Three-dimensional Dixon MRI of the pelvis were retrospectively collected for 10 patients with confirmed FAI and acquired for 10 healthy subjects. The femur and acetabulum were segmented bilaterally and associated radiomics features were extracted from the four MRI contrasts of the Dixon sequence (water-only, fat-only, in-phase, and out-of-phase). A radiologist collected 21 radiological measurements typically used in FAI. The Gini importance was used to define 9 subsets with the most predictive radiomics features and one subset for the most diagnostically relevant radiological measurements. For each subset, 100 Random Forest machine learning models were trained with different data splits and fivefold cross-validation to classify healthy subjects versus FAI patients. The average performance among the 100 models was computed for each subset and compared against the performance of the radiological measurements. One model trained using the radiomics features datasets yielded 100% accuracy in the detection of FAI, whereas all other radiomics features exceeded 80% accuracy. Radiological measurements yielded 74% accuracy, consistent with previous work. The results of this preliminary work highlight for the first time the potential of radiomics for fully automated FAI diagnosis.

Radiomic- and dosiomic-based clustering development for radio-induced neurotoxicity in pediatric medulloblastoma

BACKGROUND

Texture analysis extracts many quantitative image features, offering a valuable, cost-effective, and non-invasive approach for individual medicine. Furthermore, multimodal machine learning could have a large impact for precision medicine, as texture biomarkers can underlie tissue microstructure. This study aims to investigate imaging-based biomarkers of radio-induced neurotoxicity in pediatric patients with metastatic medulloblastoma, using radiomic and dosiomic analysis.

METHODS

This single-center study retrospectively enrolled children diagnosed with metastatic medulloblastoma (MB) and treated with hyperfractionated craniospinal irradiation (CSI). Histological confirmation of medulloblastoma and baseline follow-up magnetic resonance imaging (MRI) were mandatory. Treatment involved helical tomotherapy (HT) delivering a dose of 39 Gray (Gy) to brain and spinal axis and a posterior fossa boost up to 60 Gy. Clinical outcomes, such as local and distant brain control and neurotoxicity, were recorded. Radiomic and dosiomic features were extracted from tumor regions on T1, T2, FLAIR (fluid-attenuated inversion recovery) MRI-maps, and radiotherapy dose distribution. Different machine learning feature selection and reduction approaches were performed for supervised and unsupervised clustering.

RESULTS

Forty-eight metastatic medulloblastoma patients (29 males and 19 females) with a mean age of 12 ± 6 years were enrolled. For each patient, 332 features were extracted. Greater level of abstraction of input data by combining selection of most performing features and dimensionality reduction returns the best performance. The resulting one-component radiomic signature yielded an accuracy of 0.73 with sensitivity, specificity, and precision of 0.83, 0.64, and 0.68, respectively.

CONCLUSIONS

Machine learning radiomic-dosiomic approach effectively stratified pediatric medulloblastoma patients who experienced radio-induced neurotoxicity. Strategy needs further validation in external dataset for its potential clinical use in ab initio management paradigms of medulloblastoma.

Application of Machine Learning for Classification of Brain Tumors: A Systematic Review and Meta-Analysis

BACKGROUND

Classifying brain tumors accurately is crucial for treatment and prognosis. Machine learning (ML) shows great promise in improving tumor classification accuracy. This study evaluates ML algorithms for differentiating various brain tumor types.

METHODS

A systematic review and meta-analysis were conducted, searching PubMed, Embase, and Web of Science up to March 14, 2023. Studies that only investigated image segmentation accuracy or brain tumor detection instead of classification were excluded. We extracted binary diagnostic accuracy data, constructing contingency tables to derive sensitivity and specificity.

RESULTS

Fifty-one studies were included. The pooled area under the curve for glioblastoma versus lymphoma and low-grade versus high-grade gliomas were 0.99 (95% confidence interval [CI]: 0.98-1.00) and 0.89, respectively. The pooled sensitivity and specificity for benign versus malignant tumors were 0.90 (95% CI: 0.85-0.93) and 0.93 (95% CI: 0.90-0.95), respectively. The pooled sensitivity and specificity for low-grade versus high-grade gliomas were 0.99 (95% CI: 0.97-1.00) and 0.94, (95% CI: 0.79-0.99), respectively. Primary versus metastatic tumor identification yields sensitivity and specificity of 0.89, (95% CI: 0.83-0.93) and 0.87 (95% CI: 0.82-0.91), correspondingly. The differentiation of gliomas from pituitary tumors yielded the highest results among primary brain tumor classifications: sensitivity of 0.99 (95% CI: 0.99-1.00) and specificity of 0.99 (95% CI: 0.98-1.00).

CONCLUSIONS

ML demonstrated excellent performance in classifying brain tumor images, with near-maximum area under the curves, sensitivity, and specificity.

Improving the Generalizability and Performance of an Ultrasound Deep Learning Model Using Limited Multicenter Data for Lung Sliding Artifact Identification

Deep learning (DL) models for medical image classification frequently struggle to generalize to data from outside institutions. Additional clinical data are also rarely collected to comprehensively assess and understand model performance amongst subgroups. Following the development of a single-center model to identify the lung sliding artifact on lung ultrasound (LUS), we pursued a validation strategy using external LUS data. As annotated LUS data are relatively scarce-compared to other medical imaging data-we adopted a novel technique to optimize the use of limited external data to improve model generalizability. Externally acquired LUS data from three tertiary care centers, totaling 641 clips from 238 patients, were used to assess the baseline generalizability of our lung sliding model. We then employed our novel Threshold-Aware Accumulative Fine-Tuning (TAAFT) method to fine-tune the baseline model and determine the minimum amount of data required to achieve predefined performance goals. A subgroup analysis was also performed and Grad-CAM++ explanations were examined. The final model was fine-tuned on one-third of the external dataset to achieve 0.917 sensitivity, 0.817 specificity, and 0.920 area under the receiver operator characteristic curve (AUC) on the external validation dataset, exceeding our predefined performance goals. Subgroup analyses identified LUS characteristics that most greatly challenged the model's performance. Grad-CAM++ saliency maps highlighted clinically relevant regions on M-mode images. We report a multicenter study that exploits limited available external data to improve the generalizability and performance of our lung sliding model while identifying poorly performing subgroups to inform future iterative improvements. This approach may contribute to efficiencies for DL researchers working with smaller quantities of external validation data.

Reproducibility of radiomics quality score: an intra- and inter-rater reliability study

OBJECTIVES

To investigate the intra- and inter-rater reliability of the total radiomics quality score (RQS) and the reproducibility of individual RQS items' score in a large multireader study.

METHODS

Nine raters with different backgrounds were randomly assigned to three groups based on their proficiency with RQS utilization: Groups 1 and 2 represented the inter-rater reliability groups with or without prior training in RQS, respectively; group 3 represented the intra-rater reliability group. Thirty-three original research papers on radiomics were evaluated by raters of groups 1 and 2. Of the 33 papers, 17 were evaluated twice with an interval of 1 month by raters of group 3. Intraclass coefficient (ICC) for continuous variables, and Fleiss' and Cohen's kappa (k) statistics for categorical variables were used.

RESULTS

The inter-rater reliability was poor to moderate for total RQS (ICC 0.30-055, p < 0.001) and very low to good for item's reproducibility (k - 0.12 to 0.75) within groups 1 and 2 for both inexperienced and experienced raters. The intra-rater reliability for total RQS was moderate for the less experienced rater (ICC 0.522, p = 0.009), whereas experienced raters showed excellent intra-rater reliability (ICC 0.91-0.99, p < 0.001) between the first and second read. Intra-rater reliability on RQS items' score reproducibility was higher and most of the items had moderate to good intra-rater reliability (k - 0.40 to 1).

CONCLUSIONS

Reproducibility of the total RQS and the score of individual RQS items is low. There is a need for a robust and reproducible assessment method to assess the quality of radiomics research.

CLINICAL RELEVANCE STATEMENT

There is a need for reproducible scoring systems to improve quality of radiomics research and consecutively close the translational gap between research and clinical implementation.

KEY POINTS

• Radiomics quality score has been widely used for the evaluation of radiomics studies. • Although the intra-rater reliability was moderate to excellent, intra- and inter-rater reliability of total score and point-by-point scores were low with radiomics quality score. • A robust, easy-to-use scoring system is needed for the evaluation of radiomics research.

Machine Learning: A New Approach for Dose Individualization

The application of machine learning (ML) has shown promising results in precision medicine due to its exceptional performance in dealing with complex multidimensional data. However, using ML for individualized dosing of medicines is still in its early stage, meriting further exploration. A systematic review of study designs and modeling details of using ML for individualized dosing of different drugs was performed. We have summarized the status of the study populations, predictive targets, and data sources for ML modeling, the selection of ML algorithms and features, and the evaluation and validation of their predictive performance. We also used the Prediction model Risk of Bias Assessment Tool (PROBAST) to assess the risk of bias of included studies. Currently, ML can be used for both a priori and a posteriori dose selection and optimization, and it can also assist the implementation of therapeutic drug monitoring. However, studies are mainly focused on drugs with narrow therapeutic windows, predominantly immunosuppressants (N = 23, 35.9%) and anti-infectives (N = 21, 32.8%), and there is currently only very limited attention for special populations, such as children (N = 22, 34.4%). Most studies showed poor methodological quality and a high risk of bias. The lack of external validation and clinical utility evaluation currently limits the further clinical implementation of ML for dose individualization. We therefore have proposed several ways to improve the clinical relevance of the studies and facilitate the translation of ML models into clinical practice.

Radiomics in cone-beam breast CT for the prediction of axillary lymph node metastasis in breast cancer: a multi-center multi-device study

OBJECTIVES

To develop a radiomics model in contrast-enhanced cone-beam breast CT (CE-CBBCT) for preoperative prediction of axillary lymph node (ALN) status and metastatic burden of breast cancer.

METHODS

Two hundred and seventy-four patients who underwent CE-CBBCT examination with two scanners between 2012 and 2021 from two institutions were enrolled. The primary tumor was annotated in each patient image, from which 1781 radiomics features were extracted with PyRadiomics. After feature selection, support vector machine models were developed to predict ALN status and metastatic burden. To avoid overfitting on a specific patient subset, 100 randomly stratified splits were made to assign the patients to either training/fine-tuning or test set. Area under the receiver operating characteristic curve (AUC) of these radiomics models was compared to those obtained when training the models only with clinical features and combined clinical-radiomics descriptors. Ground truth was established by histopathology.

RESULTS

One hundred and eighteen patients had ALN metastasis (N + (≥ 1)). Of these, 74 had low burden (N + (1~2)) and 44 high burden (N + (≥ 3)). The remaining 156 patients had none (N0). AUC values across the 100 test repeats in predicting ALN status (N0/N + (≥ 1)) were 0.75 ± 0.05 (0.67~0.93, radiomics model), 0.68 ± 0.07 (0.53~0.85, clinical model), and 0.74 ± 0.05 (0.67~0.88, combined model). For metastatic burden prediction (N + (1~2)/N + (≥ 3)), AUC values were 0.65 ± 0.10 (0.50~0.88, radiomics model), 0.55 ± 0.10 (0.40~0.80, clinical model), and 0.64 ± 0.09 (0.50~0.90, combined model), with all the ranges spanning 0.5. In both cases, the radiomics model was significantly better than the clinical model (both p < 0.01) and comparable with the combined model (p = 0.56 and 0.64).

CONCLUSIONS

Radiomics features of primary tumors could have potential in predicting ALN metastasis in CE-CBBCT imaging.

CLINICAL RELEVANCE STATEMENT

The findings support potential clinical use of radiomics for predicting axillary lymph node metastasis in breast cancer patients and addressing the limited axilla coverage of cone-beam breast CT.

KEY POINTS

• Contrast-enhanced cone-beam breast CT-based radiomics could have potential to predict N0 vs. N + (≥ 1) and, to a limited extent, N + (1~2) vs. N + (≥ 3) from primary tumor, and this could help address the limited axilla coverage, pending future verifications on larger cohorts. • The average AUC of radiomics and combined models was significantly higher than that of clinical models but showed no significant difference between themselves. • Radiomics features descriptive of tumor texture were found informative on axillary lymph node status, highlighting a higher heterogeneity for tumor with positive axillary lymph node.

Classification performance bias between training and test sets in a limited mammography dataset

OBJECTIVES

To assess the performance bias caused by sampling data into training and test sets in a mammography radiomics study.

METHODS

Mammograms from 700 women were used to study upstaging of ductal carcinoma in situ. The dataset was repeatedly shuffled and split into training (n = 400) and test cases (n = 300) forty times. For each split, cross-validation was used for training, followed by an assessment of the test set. Logistic regression with regularization and support vector machine were used as the machine learning classifiers. For each split and classifier type, multiple models were created based on radiomics and/or clinical features.

RESULTS

Area under the curve (AUC) performances varied considerably across the different data splits (e.g., radiomics regression model: train 0.58-0.70, test 0.59-0.73). Performances for regression models showed a tradeoff where better training led to worse testing and vice versa. Cross-validation over all cases reduced this variability, but required samples of 500+ cases to yield representative estimates of performance.

CONCLUSIONS

In medical imaging, clinical datasets are often limited to relatively small size. Models built from different training sets may not be representative of the whole dataset. Depending on the selected data split and model, performance bias could lead to inappropriate conclusions that might influence the clinical significance of the findings.

ADVANCES IN KNOWLEDGE

Performance bias can result from model testing when using limited datasets. Optimal strategies for test set selection should be developed to ensure study conclusions are appropriate.

Machine Learning-Based Classification of Transcriptome Signatures of Non-Ulcerative Bladder Pain Syndrome

Lower urinary tract dysfunction (LUTD) presents a global health challenge with symptoms impacting a substantial percentage of the population. The absence of reliable biomarkers complicates the accurate classification of LUTD subtypes with shared symptoms such as non-ulcerative Bladder Pain Syndrome (BPS) and overactive bladder caused by bladder outlet obstruction with Detrusor Overactivity (DO). This study introduces a machine learning (ML)-based approach for the identification of mRNA signatures specific to non-ulcerative BPS. Using next-generation sequencing (NGS) transcriptome data from bladder biopsies of patients with BPS, benign prostatic obstruction with DO, and controls, our statistical approach successfully identified 13 candidate genes capable of discerning BPS from control and DO patients. This set was validated using Quantitative Polymerase Chain Reaction (QPCR) in a larger patient cohort. To confirm our findings, we applied both supervised and unsupervised ML approaches to the QPCR dataset. A three-mRNA signature TPPP3, FAT1, and NCALD, emerged as a robust classifier for non-ulcerative BPS. The ML-based framework used to define BPS classifiers establishes a solid foundation for comprehending the gene expression changes in the bladder during BPS and serves as a valuable resource and methodology for advancing signature identification in other fields. The proposed ML pipeline demonstrates its efficacy in handling challenges associated with limited sample sizes, offering a promising avenue for applications in similar domains.

Automatic segmentation of ameloblastoma on ct images using deep learning with limited data

BACKGROUND

Ameloblastoma, a common benign tumor found in the jaw bone, necessitates accurate localization and segmentation for effective diagnosis and treatment. However, the traditional manual segmentation method is plagued with inefficiencies and drawbacks. Hence, the implementation of an AI-based automatic segmentation approach is crucial to enhance clinical diagnosis and treatment procedures.

METHODS

We collected CT images from 79 patients diagnosed with ameloblastoma and employed a deep learning neural network model for training and testing purposes. Specifically, we utilized the Mask R-CNN neural network structure and implemented image preprocessing and enhancement techniques. During the testing phase, cross-validation methods were employed for evaluation, and the experimental results were verified using an external validation set. Finally, we obtained an additional dataset comprising 200 CT images of ameloblastoma from a different dental center to evaluate the model's generalization performance.

RESULTS

During extensive testing and evaluation, our model successfully demonstrated the capability to automatically segment ameloblastoma. The DICE index achieved an impressive value of 0.874. Moreover, when the IoU threshold ranged from 0.5 to 0.95, the model's AP was 0.741. For a specific IoU threshold of 0.5, the model achieved an AP of 0.914, and for another IoU threshold of 0.75, the AP was 0.826. Our validation using external data confirms the model's strong generalization performance.

CONCLUSION

In this study, we successfully applied a neural network model based on deep learning that effectively performs automatic segmentation of ameloblastoma. The proposed method offers notable advantages in terms of efficiency, accuracy, and speed, rendering it a promising tool for clinical diagnosis and treatment.

Machine Learning-based Classification of transcriptome Signatures of non-ulcerative Bladder Pain Syndrome

Lower urinary tract dysfunction (LUTD) presents a global health challenge with symptoms impacting a substantial percentage of the population. The absence of reliable biomarkers complicates the accurate classification of LUTD subtypes with shared symptoms such as non-ulcerative Bladder Pain Syndrome (BPS) and overactive bladder caused by bladder outlet obstruction with Detrusor Overactivity (DO). This study introduces a machine learning (ML)-based approach for the identification of mRNA signatures specific to non-ulcerative BPS. Using next-generation sequencing (NGS) transcriptome data from bladder biopsies of patients with BPS, benign prostatic obstruction with DO and controls, our statistical approach successfully identified 13 candidate genes capable of discerning BPS from control and DO patients. This set was subsequently validated using Quantitative Polymerase Chain Reaction (QPCR) in a larger patient cohort. To confirm our findings, we applied both supervised and unsupervised ML approaches to the QPCR dataset. Notably, a three-mRNA signature TPPP3, FAT1, and NCALD, emerged as a robust classifier, effectively distinguishing patients with non-ulcerative BPS from controls and patients with DO. This signature was universally selected by both supervised and unsupervised approaches. The ML-based framework used to define BPS classifiers not only establishes a solid foundation for comprehending the specific gene expression changes in the bladder of the patients with BPS but also serves as a valuable resource and methodology for advancing signature identification in other fields. The proposed ML pipeline demonstrates its efficacy in handling challenges associated with limited sample sizes, offering a promising avenue for applications in similar domains.

Radiomics: The New Promise for Differentiating Progression, Recurrence, Pseudoprogression, and Radionecrosis in Glioma and Glioblastoma Multiforme

Glioma and glioblastoma multiform (GBM) remain among the most debilitating and life-threatening brain tumors. Despite advances in diagnosing approaches, patient follow-up after treatment (surgery and chemoradiation) is still challenging for differentiation between tumor progression/recurrence, pseudoprogression, and radionecrosis. Radiomics emerges as a promising tool in initial diagnosis, grading, and survival prediction in patients with glioma and can help differentiate these post-treatment scenarios. Preliminary published studies are promising about the role of radiomics in post-treatment glioma/GBM. However, this field faces significant challenges, including a lack of evidence-based solid data, scattering publication, heterogeneity of studies, and small sample sizes. The present review explores radiomics's capabilities in following patients with glioma/GBM status post-treatment and to differentiate tumor progression, recurrence, pseudoprogression, and radionecrosis.

Comparative performances of machine learning algorithms in radiomics and impacting factors

There are no current recommendations on which machine learning (ML) algorithms should be used in radiomics. The objective was to compare performances of ML algorithms in radiomics when applied to different clinical questions to determine whether some strategies could give the best and most stable performances regardless of datasets. This study compares the performances of nine feature selection algorithms combined with fourteen binary classification algorithms on ten datasets. These datasets included radiomics features and clinical diagnosis for binary clinical classifications including COVID-19 pneumonia or sarcopenia on CT, head and neck, orbital or uterine lesions on MRI. For each dataset, a train-test split was created. Each of the 126 (9 × 14) combinations of feature selection algorithms and classification algorithms was trained and tuned using a ten-fold cross validation, then AUC was computed. This procedure was repeated three times per dataset. Best overall performances were obtained with JMI and JMIM as feature selection algorithms and random forest and linear regression models as classification algorithms. The choice of the classification algorithm was the factor explaining most of the performance variation (10% of total variance). The choice of the feature selection algorithm explained only 2% of variation, while the train-test split explained 9%.

Radiomics-Clinical AI Model with Probability Weighted Strategy for Prognosis Prediction in Non-Small Cell Lung Cancer

In this study, we propose a radiomics clinical probability-weighted model for the prediction of prognosis for non-small cell lung cancer (NSCLC). The model combines radiomics features extracted from radiotherapy (RT) planning images with clinical factors such as age, gender, histology, and tumor stage. CT images with radiotherapy structures of 422 NSCLC patients were retrieved from The Cancer Imaging Archive (TCIA). Radiomic features were extracted from gross tumor volumes (GTVs). Five machine learning algorithms, namely decision trees (DT), random forests (RF), extreme boost (EB), support vector machine (SVM) and generalized linear model (GLM) were optimized by a voted ensemble machine learning (VEML) model. A probabilistic weighted approach is used to incorporate the uncertainty associated with both radiomic and clinical features and to generate a probabilistic risk score for each patient. The performance of the model is evaluated using a receiver operating characteristic (ROC). The Radiomic model, clinical factor model, and combined radiomic clinical probability-weighted model demonstrated good performance in predicting NSCLC survival with AUC of 0.941, 0.856 and 0.949, respectively. The combined radiomics clinical probability-weighted enhanced model achieved significantly better performance than the radiomic model in 1-year survival prediction (chi-square test, p < 0.05). The proposed model has the potential to improve NSCLC prognosis and facilitate personalized treatment decisions.

Brain imaging signatures of neuropathic facial pain derived by artificial intelligence

Advances in neuroimaging have permitted the non-invasive examination of the human brain in pain. However, a persisting challenge is in the objective differentiation of neuropathic facial pain subtypes, as diagnosis is based on patients' symptom descriptions. We use artificial intelligence (AI) models with neuroimaging data to distinguish subtypes of neuropathic facial pain and differentiate them from healthy controls. We conducted a retrospective analysis of diffusion tensor and T1-weighted imaging data using random forest and logistic regression AI models on 371 adults with trigeminal pain (265 classical trigeminal neuralgia (CTN), 106 trigeminal neuropathic pain (TNP)) and 108 healthy controls (HC). These models distinguished CTN from HC with up to 95% accuracy, and TNP from HC with up to 91% accuracy. Both classifiers identified gray and white matter-based predictive metrics (gray matter thickness, surface area, and volume; white matter diffusivity metrics) that significantly differed across groups. Classification of TNP and CTN did not show significant accuracy (51%) but highlighted two structures that differed between pain groups-the insula and orbitofrontal cortex. Our work demonstrates that AI models with brain imaging data alone can differentiate neuropathic facial pain subtypes from healthy data and identify regional structural indicates of pain.

Radiomics and dosiomics-based prediction of radiotherapy-induced xerostomia in head and neck cancer patients

BACKGROUND AND AIM

Dose-response modeling for radiotherapy-induced xerostomia in head and neck cancer (HN) patients is a promising frontier for personalized therapy. Feature extraction from diagnostic and therapeutic images (radiomics and dosiomics features) can be used for data-driven response modeling. The aim of this study is to develop xerostomia predictive models based on radiomics-dosiomics features.

METHODS

Data from the cancer imaging archive (TCIA) for 31 HN cancer patients were employed. For all patients, parotid CT radiomics features were extracted, utilizing Lasso regression for feature selection and multivariate modeling. The models were developed by selected features from pretreatment (CT1), mid-treatment (CT2), post-treatment (CT3), and delta features (ΔCT2-1, ΔCT3-1, ΔCT3-2). We also considered dosiomics features extracted from the parotid dose distribution images (Dose model). Thus, combination models of radio-dosiomics (CT + dose & ΔCT + dose) were developed. Moreover, clinical, and dose-volume histogram (DVH) models were built. Nested 10-fold cross-validation was used to assess the predictive classification of patients into those with and without xerostomia, and the area under the receiver operative characteristic curve (AUC) was used to compare the predictive power of the models. The sensitivity and accuracy of models also were obtained.

RESULTS

In total, 59 parotids were assessed, and 13 models were developed. Our results showed three models with AUC of 0.89 as most predictive, namely ΔCT2-1 + Dose (Sensitivity 0.99, Accuracy 0.94 & Specificity 0.86), CT3 model (Sensitivity 0.96, Accuracy 0.94 & Specificity 0.86) and DVH (Sensitivity 0.93, Accuracy 0.89 & Specificity 0.84). These models were followed by Clinical (AUC 0.89, Sensitivity 0.81, Accuracy 0.97 & Specificity 0.89) and CT2 & Dose (AUC 0.86, Sensitivity 0.97, Accuracy 0.87 & Specificity 0.82). The Dose model (developed by dosiomics features only) had AUC, Sensitivity, Specificity, and Accuracy of 0.72, 0.98, 0.33, and 0.79 respectively.

CONCLUSION

Quantitative features extracted from diagnostic imaging during and after radiotherapy alone or in combination with dosiomics markers obtained from dose distribution images can be used for radiotherapy response modeling, opening up prospects for personalization of therapies toward improved therapeutic outcomes.

Must-have Qualities of Clinical Research on Artificial Intelligence and Machine Learning

In the field of computer science, known as artificial intelligence, algorithms imitate reasoning tasks that are typically performed by humans. The techniques that allow machines to learn and get better at tasks such as recognition and prediction, which form the basis of clinical practice, are referred to as machine learning, which is a subfield of artificial intelligence. The number of artificial intelligence-and machine learnings-related publications in clinical journals has grown exponentially, driven by recent developments in computation and the accessibility of simple tools. However, clinicians are often not included in data science teams, which may limit the clinical relevance, explanability, workflow compatibility, and quality improvement of artificial intelligence solutions. Thus, this results in the language barrier between clinicians and artificial intelligence developers. Healthcare practitioners sometimes lack a basic understanding of artificial intelligence research because the approach is difficult for non-specialists to understand. Furthermore, many editors and reviewers of medical publications might not be familiar with the fundamental ideas behind these technologies, which may prevent journals from publishing high-quality artificial intelligence studies or, worse still, could allow for the publication of low-quality works. In this review, we aim to improve readers’ artificial intelligence literacy and critical thinking. As a result, we concentrated on what we consider the 10 most important qualities of artificial intelligence research: valid scientific purpose, high-quality data set, robust reference standard, robust input, no information leakage, optimal bias-variance tradeoff, proper model evaluation, proven clinical utility, transparent reporting, and open science. Before designing a study, one should have defined a sound scientific purpose. Then, it should be backed by a high-quality data set, robust input, and a solid reference standard. The artificial intelligence development pipeline should prevent information leakage. For the models, optimal bias-variance tradeoff should be achieved, and generalizability assessment must be adequately performed. The clinical value of the final models must also be established. After the study, thought should be given to transparency in publishing the process and results as well as open science for sharing data, code, and models. We hope this work may improve the artificial intelligence literacy and mindset of the readers.

Radiomic features of amygdala nuclei and hippocampus subfields help to predict subthalamic deep brain stimulation motor outcomes for Parkinson‘s disease patients

Background and purpose

The aim of the study is to predict the subthalamic nucleus (STN) deep brain stimulation (DBS) outcomes for Parkinson’s disease (PD) patients using the radiomic features extracted from pre-operative magnetic resonance images (MRI).

Methods

The study included 34 PD patients who underwent DBS implantation in the STN. Five patients (15%) showed poor DBS motor outcome. All together 9 amygdalar nuclei and 12 hippocampus subfields were segmented using Freesurfer 7.0 pipeline from pre-operative MRI images. Furthermore, PyRadiomics platform was used to extract 120 radiomic features for each nuclei and subfield resulting in 5,040 features. Minimum Redundancy Maximum Relevance (mRMR) feature selection method was employed to reduce the number of features to 20, and 8 machine learning methods (regularized binary logistic regression (LR), decision tree classifier (DT), linear discriminant analysis (LDA), naive Bayes classifier (NB), kernel support vector machine (SVM), deep feed-forward neural network (DNN), one-class support vector machine (OC-SVM), feed-forward neural network-based autoencoder for anomaly detection (DNN-A)) were applied to build the models for poor vs. good and very good STN-DBS motor outcome prediction.

Results

The highest mean prediction accuracy was obtained using regularized LR (96.65 ± 7.24%, AUC 0.98 ± 0.06) and DNN (87.25 ± 14.80%, AUC 0.87 ± 0.18).

Conclusion

The results show the potential power of the radiomic features extracted from hippocampus and amygdala MRI in the prediction of STN-DBS motor outcomes for PD patients.

Validation of MRI-Based Models to Predict MGMT Promoter Methylation in Gliomas: BraTS 2021 Radiogenomics Challenge

Simple Summary

O6-methylguanine-DNA methyl transferase (MGMT) methylation in glioblastoma is an important prognostic and predictive factor that requires an invasive surgical procedure for identification. In several recent studies, MGMT methylation prediction models were developed using MR images, and good diagnostic performance was achieved, which seems to indicate a promising future for radiogenomics. However, the diagnostic performance was not reproducible for numerous research teams when using a larger dataset in the RSNA-MICCAI Brain Tumor Radiogenomic Classification 2021 challenge. To our knowledge, there has been no study regarding the external validation of MGMT prediction models using large-scale multicenter datasets. We tested recent CNN architectures via extensive experiments to investigate whether MGMT methylation in gliomas can be predicted using MRI. With unexpected negative results, approximately 80% of the developed models showed no significant difference with the chance level of 50% in terms of external validation accuracy. In conclusion, MGMT methylation status of gliomas may not be predictable with preoperative MRI, even using deep learning.

Abstract

O6-methylguanine-DNA methyl transferase (MGMT) methylation prediction models were developed using only small datasets without proper external validation and achieved good diagnostic performance, which seems to indicate a promising future for radiogenomics. However, the diagnostic performance was not reproducible for numerous research teams when using a larger dataset in the RSNA-MICCAI Brain Tumor Radiogenomic Classification 2021 challenge. To our knowledge, there has been no study regarding the external validation of MGMT prediction models using large-scale multicenter datasets. We tested recent CNN architectures via extensive experiments to investigate whether MGMT methylation in gliomas can be predicted using MR images. Specifically, prediction models were developed and validated with different training datasets: (1) the merged (SNUH + BraTS) (n = 985); (2) SNUH (n = 400); and (3) BraTS datasets (n = 585). A total of 420 training and validation experiments were performed on combinations of datasets, convolutional neural network (CNN) architectures, MRI sequences, and random seed numbers. The first-place solution of the RSNA-MICCAI radiogenomic challenge was also validated using the external test set (SNUH). For model evaluation, the area under the receiver operating characteristic curve (AUROC), accuracy, precision, and recall were obtained. With unexpected negative results, 80.2% (337/420) and 60.0% (252/420) of the 420 developed models showed no significant difference with a chance level of 50% in terms of test accuracy and test AUROC, respectively. The test AUROC and accuracy of the first-place solution of the BraTS 2021 challenge were 56.2% and 54.8%, respectively, as validated on the SNUH dataset. In conclusion, MGMT methylation status of gliomas may not be predictable with preoperative MR images even using deep learning.

Can Sequential Images from the Same Object Be Used for Training Machine Learning Models? A Case Study for Detecting Liver Disease by Ultrasound Radiomics

Machine learning for medical imaging not only requires sufficient amounts of data for training and testing but also that the data be independent. It is common to see highly interdependent data whenever there are inherent correlations between observations. This is especially to be expected for sequential imaging data taken from time series. In this study, we evaluate the use of statistical measures to test the independence of sequential ultrasound image data taken from the same case. A total of 1180 B-mode liver ultrasound images with 5903 regions of interests were analyzed. The ultrasound images were taken from two liver disease groups, fibrosis and steatosis, as well as normal cases. Computer-extracted texture features were then used to train a machine learning (ML) model for computer-aided diagnosis. The experiment resulted in high two-category diagnosis using logistic regression, with AUC of 0.928 and high performance of multicategory classification, using random forest ML, with AUC of 0.917. To evaluate the image region independence for machine learning, Jenson–Shannon (JS) divergence was used. JS distributions showed that images of normal liver were independent from each other, while the images from the two disease pathologies were not independent. To guarantee the generalizability of machine learning models, and to prevent data leakage, multiple frames of image data acquired of the same object should be tested for independence before machine learning. Such tests can be applied to real-world medical image problems to determine if images from the same subject can be used for training.

Radiomics software comparison using digital phantom and patient data: IBSI-compliance does not guarantee concordance of feature values

Introduction Radiomics is a promising imaging-based tool which could enhance clinical observation and identify representative features. To avoid different interpretations, the Image Biomarker Standardisation Initiative (IBSI) imposed conditions for harmonisation. This study evaluates IBSI-compliant radiomics applications against a known benchmark and clinical datasets for agreements. Materials and methods The applications compared were RadiomiX Research Toolbox, LIFEx v7.0.0, and syngo.via Frontier Radiomics v1.2.5 (based on PyRadiomics v2.1). Basic assessment included comparing feature names and their formulas. The IBSI digital phantom was used for evaluation against reference values. For agreement evaluation (including same software but different versions), two clinical datasets were used: 27 contrast-enhanced computed tomography (CECT) of colorectal liver metastases and 39 magnetic resonance imaging (MRI) of breast cancer, including intravoxel incoherent motion and dynamic contrast-enhanced (DCE) MRI. The lower 95% confidence interval of intraclass correlation coefficient was used to define excellent agreement (>0.9). Results The three radiomics applications share 41 (3 shape, 8 intensity, 30 texture) out of 172, 84 and 110 features for RadiomiX, LIFEx and syngo.via, respectively, as well as wavelet filtering. The naming convention is, however, different between them. Syngo.via had excellent agreement with the IBSI benchmark, while LIFEx and RadiomiX showed slightly worse agreement. Excellent reproducibility was achieved for shape features only, while intensity and texture features varied considerably with the imaging type. For intensity, excellent agreement ranged from 46% for the DCE maps to 100% for CECT, while this lowered to 44% and 73% for texture features, respectively. Wavelet features produced the greatest variation between applications, with an excellent agreement for only 3% to 11% features. Conclusion Even with IBSI-compliance, the reproducibility of features between radiomics applications is not guaranteed. To evaluate variation, quality assurance of radiomics applications should be performed and repeated when updating to a new version or adding a new modality.

Prediction of 5-year progression-free survival in advanced nasopharyngeal carcinoma with pretreatment PET/CT using multi-modality deep learning-based radiomics

Objective

Deep learning-based radiomics (DLR) has achieved great success in medical image analysis and has been considered a replacement for conventional radiomics that relies on handcrafted features. In this study, we aimed to explore the capability of DLR for the prediction of 5-year progression-free survival (PFS) in advanced nasopharyngeal carcinoma (NPC) using pretreatment PET/CT images.

Methods

A total of 257 patients (170/87 patients in internal/external cohorts) with advanced NPC (TNM stage III or IVa) were enrolled. We developed an end-to-end multi-modality DLR model, in which a 3D convolutional neural network was optimized to extract deep features from pretreatment PET/CT images and predict the probability of 5-year PFS. The TNM stage, as a high-level clinical feature, could be integrated into our DLR model to further improve the prognostic performance. For a comparison between conventional radiomics and DLR, 1,456 handcrafted features were extracted, and optimal conventional radiomics methods were selected from 54 cross-combinations of six feature selection methods and nine classification methods. In addition, risk group stratification was performed with clinical signature, conventional radiomics signature, and DLR signature.

Results

Our multi-modality DLR model using both PET and CT achieved higher prognostic performance (area under the receiver operating characteristic curve (AUC) = 0.842 ± 0.034 and 0.823 ± 0.012 for the internal and external cohorts) than the optimal conventional radiomics method (AUC = 0.796 ± 0.033 and 0.782 ± 0.012). Furthermore, the multi-modality DLR model outperformed single-modality DLR models using only PET (AUC = 0.818 ± 0.029 and 0.796 ± 0.009) or only CT (AUC = 0.657 ± 0.055 and 0.645 ± 0.021). For risk group stratification, the conventional radiomics signature and DLR signature enabled significant difference between the high- and low-risk patient groups in both the internal and external cohorts (p < 0.001), while the clinical signature failed in the external cohort (p = 0.177).

Conclusion

Our study identified potential prognostic tools for survival prediction in advanced NPC, which suggests that DLR could provide complementary values to the current TNM staging.

Adding radiomics to the 2021 WHO updates may improve prognostic prediction for current IDH-wildtype histological lower-grade gliomas with known EGFR amplification and TERT promoter mutation status

OBJECTIVES

To assess whether radiomic features could improve the accuracy of survival predictions of IDH-wildtype (IDHwt) histological lower-grade gliomas (LGGs) over clinicopathological features.

METHODS

Preoperative MRI data of 61 patients with IDHwt histological LGGs were included as the institutional training set. The test set consisted of 32 patients from The Cancer Genome Atlas. Radiomic features (n = 186) were extracted using conventional MRIs. The radiomics risk score (RRS) for overall survival (OS) was derived from the elastic net. Multivariable Cox regression analyses with clinicopathological features (including epidermal growth factor receptor [EGFR] amplification and telomerase reverse transcriptase promoter [TERTp] mutation status) and the RRS were performed. The integrated area under the receiver operating curves (iAUCs) from the models with and without the RRS were compared. The net reclassification index (NRI) for 1-year OS was also calculated. The prognostic value of the RRS was evaluated using the external validation set.

RESULTS

The RRS independently predicted OS (hazard ratio = 48.08; p = 0.001). Compared with the clinicopathological model alone, adding the RRS had a better OS prediction performance (iAUCs 0.775 vs. 0.910), which was internally validated (iAUCs 0.726 vs. 0.884, 1-year OS NRI = 0.497), and a similar trend was found on external validation (iAUCs 0.683 vs. 0.705, 1-year OS NRI = 0.733). The prognostic significance of the RRS was confirmed in the external validation set (p = 0.001).

CONCLUSIONS

Integrating radiomics with clinicopathological features (including EGFR amplification and TERTp mutation status) can improve survival prediction in patients with IDHwt LGGs.

KEY POINTS

• Radiomics risk score has the potential to improve survival prediction when added to clinicopathological features (iAUCs increased from 0.775 to 0.910). • NRIs for 1-year OS showed that the radiomics risk score had incremental value over the clinicopathological model. • The prognostic significance of the radiomics risk score was confirmed in the external validation set (p = 0.001).

Machine-learning approach to predict molecular subgroups of medulloblastoma using multiparametric MRI-based tumor radiomics

Objective:

Image-based prediction of molecular subgroups of Medulloblastoma (MB) has the potential to optimize and personalize therapy. The objective of the study is to distinguish between broad molecular subgroups of MB using MR–Texture analysis.

Methods:

Thirty-eight MB patients treated between 2007 and 2020 were retrospectively analyzed. Texture analysis was performed on contrast enhanced T1(T1C) and T2 weighted (T2W) MR images. Manual segmentation was performed on all slices and radiomic features were extracted which included first order, second order (GLCM - Grey level co-occurrence matrix) and shape features. Feature enrichment was done using LASSO (Least Absolute Shrinkage and Selection Operator) regression and thereafter Support Vector Machine (SVM) and a 10-fold cross-validation strategy was used for model development. The area under Receiver Operator Characteristic (ROC) curve was used to evaluate the model.

Results:

A total of 174 and 170 images were obtained for analysis from the Axial T1C and T2W image datasets. One hundred and sixty-four MR based texture features were extracted. The best model was arrived at by using a combination of 30 GLCM and six shape features on T1C MR sequence. A 10-fold cross-validation demonstrated an AUC of 0.93, 0.9, 0.93, and 0.93 in predicting WNT, SHH, Group 3, and Group 4 MB subgroups, respectively.

Conclusion:

Radiomic analysis of MR images in MB can predict molecular subgroups with acceptable degree of accuracy. The strategy needs further validation in an external dataset for its potential use in ab initio management paradigms of MBs.

Advances in knowledge:

Medulloblastoma can be classified into four distinct molecular subgroups using radiomic feature classifier from non-invasive Multiparametric Magnetic resonance imaging. This can have future ramifications in the extent of surgical resection of Medulloblastoma which can ultimately result in reduction of morbidity.

Mortality Prediction of COVID-19 Patients Using Radiomic and Neural Network Features Extracted from a Wide Chest X-ray Sample Size: A Robust Approach for Different Medical Imbalanced Scenarios

Aim: The aim of this study was to develop robust prognostic models for mortality prediction of COVID-19 patients, applicable to different sets of real scenarios, using radiomic and neural network features extracted from chest X-rays (CXRs) with a certified and commercially available software. Methods: 1816 patients from 5 different hospitals in the Province of Reggio Emilia were included in the study. Overall, 201 radiomic features and 16 neural network features were extracted from each COVID-19 patient’s radiography. The initial dataset was balanced to train the classifiers with the same number of dead and survived patients, randomly selected. The pipeline had three main parts: balancing procedure; three-step feature selection; and mortality prediction with radiomic features through three machine learning (ML) classification models: AdaBoost (ADA), Quadratic Discriminant Analysis (QDA) and Random Forest (RF). Five evaluation metrics were computed on the test samples. The performance for death prediction was validated on both a balanced dataset (Case 1) and an imbalanced dataset (Case 2). Results: accuracy (ACC), area under the ROC-curve (AUC) and sensitivity (SENS) for the best classifier were, respectively, 0.72 ± 0.01, 0.82 ± 0.02 and 0.84 ± 0.04 for Case 1 and 0.70 ± 0.04, 0.79 ± 0.03 and 0.76 ± 0.06 for Case 2. These results show that the prediction of COVID-19 mortality is robust in a different set of scenarios. Conclusions: Our large and varied dataset made it possible to train ML algorithms to predict COVID-19 mortality using radiomic and neural network features of CXRs.

Radiomics Based on Digital Mammography Helps to Identify Mammographic Masses Suspicious for Cancer

Objectives

This study aims to build radiomics model of Breast Imaging Reporting and Data System (BI-RADS) category 4 and 5 mammographic masses extracted from digital mammography (DM) for mammographic masses characterization by using a sensitivity threshold similar to that of biopsy.

Materials and Methods

This retrospective study included 288 female patients (age, 52.41 ± 10.31) who had BI-RADS category 4 or 5 mammographic masses with an indication for biopsy. The patients were divided into two temporal set (training set, 82 malignancies and 110 benign lesions; independent test set, 48 malignancies and 48 benign lesions). A total of 188 radiomics features were extracted from mammographic masses on the combination of craniocaudal (CC) position images and mediolateral oblique (MLO) position images. For the training set, Pearson’s correlation and the least absolute shrinkage and selection operator (LASSO) were used to select non-redundant radiomics features and useful radiomics features, respectively, and support vector machine (SVM) was applied to construct a radiomics model. The receiver operating characteristic curve (ROC) analysis was used to evaluate the classification performance of the radiomics model and to determine a threshold value with a sensitivity higher than 98% to predict the mammographic masses malignancy. For independent test set, identical threshold value was used to validate the classification performance of the radiomics model. The stability of the radiomics model was evaluated by using a fivefold cross-validation method, and two breast radiologists assessed the diagnostic agreement of the radiomics model.

Results

In the training set, the radiomics model obtained an area under the receiver operating characteristic curve (AUC) of 0.934 [95% confidence intervals (95% CI), 0.898–0.971], a sensitivity of 98.8% (81/82), a threshold of 0.22, and a specificity of 60% (66/110). In the test set, the radiomics model obtained an AUC of 0.901 (95% CI, 0.835–0.961), a sensitivity of 95.8% (46/48), and a specificity of 66.7% (32/48). The radiomics model had relatively stable sensitivities in fivefold cross-validation (training set, 97.39% ± 3.9%; test set, 98.7% ± 4%).

Conclusion

The radiomics method based on DM may help reduce the temporarily unnecessary invasive biopsies for benign mammographic masses over-classified in BI-RADS category 4 and 5 while providing similar diagnostic performance for malignant mammographic masses as biopsies.

A fully automatic multiparametric radiomics model for differentiation of adult pilocytic astrocytomas from glioblastomas

OBJECTIVES

To develop a fully automatic radiomics model to differentiate adult pilocytic astrocytomas (PA) from high-grade gliomas (HGGs).

METHODS

This retrospective study included 302 adult patients with PA (n = 62) or HGG (n = 240). The patients were randomly divided into training (n = 211) and test (n = 91) sets. Clinical data were obtained, and radiomic features (n = 372) were extracted from multiparametric MRI with automatic tumour segmentation. After feature selection with F-score, a Light Gradient Boosting Machine classifier with subsampling was trained to develop three models: (1) clinical model, (2) radiomics model, and (3) combined clinical and radiomics model. Human performance was also assessed. The performance of the classifier was validated in the test set. SHapley Additive exPlanations (SHAP) was applied to explore the interpretability of the model.

RESULTS

A total of 15 radiomic features were selected. In the test set, the combined clinical and radiomics model (area under the curve [AUC], 0.93) showed a significantly higher performance than the clinical model (AUC, 0.79, p = 0.037) and had a similar performance to the radiomics model (AUC, 0.92, p = 0.828). The combined clinical and radiomics model also showed a significantly higher performance than humans (AUC, 0.76-0.81, p < 0.05). The model explanation by SHAP suggested that lower intratumoural heterogeneity from T2-weighted images was highly associated with PA diagnosis.

CONCLUSIONS

The fully automatic combined clinical and radiomics model may be helpful for differentiating adult PAs from HGGs.

KEY POINTS

• Differentiating adult PAs from HGGs is challenging because PAs may manifest a large spectrum of imaging presentations, often including aggressive imaging features. • The fully automatic combined clinical and radiomics model showed a significantly higher performance than the clinical model or humans. • The model explanation by SHAP suggested that second-order features from T2-weighted imaging were important in diagnosis and might reflect the underlying pathophysiology that PAs have lesser tissue heterogeneity than HGGs.