Quality assessment of radiomics research in cardiac CT: a systematic review

Methodological quality of radiomic-based prognostic studies in gastric cancer: a cross-sectional study

Background

Machine learning radiomics models are increasingly being used to predict gastric cancer prognoses. However, the methodological quality of these models has not been evaluated. Therefore, this study aimed to evaluate the methodological quality of radiomics studies in predicting the prognosis of gastric cancer, summarize their methodological characteristics and performance.

Methods

The PubMed and Embase databases were searched for radiomics studies used to predict the prognosis of gastric cancer published in last 5 years. The characteristics of the studies and the performance of the models were extracted from the eligible full texts. The methodological quality, reporting completeness and risk of bias of the included studies were evaluated using the RQS, TRIPOD and PROBAST. The discrimination ability scores of the models were also compared.

Results

Out of 283 identified records, 22 studies met the inclusion criteria. The study endpoints included survival time, treatment response, and recurrence, with reported discriminations ranging between 0.610 and 0.878 in the validation dataset. The mean overall RQS value was 15.32 ± 3.20 (range: 9 to 21). The mean adhered items of the 35 item of TRIPOD checklist was 20.45 ± 1.83. The PROBAST showed all included studies were at high risk of bias.

Conclusion

The current methodological quality of gastric cancer radiomics studies is insufficient. Large and reasonable sample, prospective, multicenter and rigorously designed studies are required to improve the quality of radiomics models for gastric cancer prediction.

Study registration

This protocol was prospectively registered in the Open Science Framework Registry (https://osf.io/ja52b).

A systematic review of radiomics in chondrosarcoma: assessment of study quality and clinical value needs handy tools

OBJECTIVE

To evaluate the study quality and clinical value of radiomics studies on chondrosarcoma.

METHODS

PubMed, Embase, Web of Science, China National Knowledge Infrastructure, and Wanfang Data were searched for articles on radiomics for evaluating chondrosarcoma as of January 31, 2022. The study quality was assessed according to Radiomics Quality Score (RQS), Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) checklist, Image Biomarker Standardization Initiative (IBSI) guideline, and modified Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2) tool. The level of evidence supporting clinical use of radiomics on chondrosarcoma differential diagnosis was determined based on meta-analyses.

RESULTS

Twelve articles were included. The median RQS was 10.5 (range, -3 to 15), with an adherence rate of 36%. The adherence rate was extremely low in domains of high-level evidence (0%), open science and data (17%), and imaging and segmentation (35%). The adherence rate of the TRIPOD checklist was 61%, and low for section of title and abstract (13%), introduction (42%), and results (56%). The reporting rate of pre-processing steps according to the IBSI guideline was 60%. The risk of bias and concern of application were mainly related to the index test. The meta-analysis on differential diagnosis of enchondromas vs. chondrosarcomas showed a diagnostic odds ratio of 43.90 (95% confidential interval, 25.33-76.10), which was rated as weak evidence.

CONCLUSIONS

The current scientific and reporting quality of radiomics studies on chondrosarcoma was insufficient. Radiomics has potential in facilitating the optimization of operation decision-making in chondrosarcoma.

KEY POINTS

• Among radiomics studies on chondrosarcoma, although differential diagnostic models showed promising performance, only pieces of weak level of evidence were reached with insufficient study quality. • Since the RQS rating, the TRIPOD checklist, and the IBSI guideline have largely overlapped with each other, it is necessary to establish one widely acceptable methodological and reporting guideline for radiomics research. • The TRIPOD model typing, the phase classification of image mining studies, and the level of evidence category are useful tools to assess the gap between academic research and clinical application, although their modifications for radiomics studies are needed.

Diagnosis of Idiopathic Pulmonary Fibrosis in High-Resolution Computed Tomography Scans Using a Combination of Handcrafted Radiomics and Deep Learning

Purpose

To develop handcrafted radiomics (HCR) and deep learning (DL) based automated diagnostic tools that can differentiate between idiopathic pulmonary fibrosis (IPF) and non-IPF interstitial lung diseases (ILDs) in patients using high-resolution computed tomography (HRCT) scans.

Material and Methods

In this retrospective study, 474 HRCT scans were included (mean age, 64.10 years ± 9.57 [SD]). Five-fold cross-validation was performed on 365 HRCT scans. Furthermore, an external dataset comprising 109 patients was used as a test set. An HCR model, a DL model, and an ensemble of HCR and DL model were developed. A virtual in-silico trial was conducted with two radiologists and one pulmonologist on the same external test set for performance comparison. The performance was compared using DeLong method and McNemar test. Shapley Additive exPlanations (SHAP) plots and Grad-CAM heatmaps were used for the post-hoc interpretability of HCR and DL models, respectively.

Results

In five-fold cross-validation, the HCR model, DL model, and the ensemble of HCR and DL models achieved accuracies of 76.2 ± 6.8, 77.9 ± 4.6, and 85.2 ± 2.7%, respectively. For the diagnosis of IPF and non-IPF ILDs on the external test set, the HCR, DL, and the ensemble of HCR and DL models achieved accuracies of 76.1, 77.9, and 85.3%, respectively. The ensemble model outperformed the diagnostic performance of clinicians who achieved a mean accuracy of 66.3 ± 6.7% (p < 0.05) during the in-silico trial. The area under the receiver operating characteristic curve (AUC) for the ensemble model on the test set was 0.917 which was significantly higher than the HCR model (0.817, p = 0.02) and the DL model (0.823, p = 0.005). The agreement between HCR and DL models was 61.4%, and the accuracy and specificity for the predictions when both the models agree were 93 and 97%, respectively. SHAP analysis showed the texture features as the most important features for IPF diagnosis and Grad-CAM showed that the model focused on the clinically relevant part of the image.

Conclusion

Deep learning and HCR models can complement each other and serve as useful clinical aids for the diagnosis of IPF and non-IPF ILDs.