An updated systematic review of radiomics in osteosarcoma: utilizing CLAIM to adapt the increasing trend of deep learning application in radiomics

Self-reporting with checklists in artificial intelligence research on medical imaging: a systematic review based on citations of CLAIM

OBJECTIVE

To evaluate the usage of a well-known and widely adopted checklist, Checklist for Artificial Intelligence in Medical imaging (CLAIM), for self-reporting through a systematic analysis of its citations.

METHODS

Google Scholar, Web of Science, and Scopus were used to search for citations (date, 29 April 2023). CLAIM's use for self-reporting with proof (i.e., filled-out checklist) and other potential use cases were systematically assessed in research papers. Eligible papers were evaluated independently by two readers, with the help of automatic annotation. Item-by-item confirmation analysis on papers with checklist proof was subsequently performed.

RESULTS

A total of 391 unique citations were identified from three databases. Of the 118 papers included in this study, 12 (10%) provided a proof of self-reported CLAIM checklist. More than half (70; 59%) only mentioned some sort of adherence to CLAIM without providing any proof in the form of a checklist. Approximately one-third (36; 31%) cited the CLAIM for reasons unrelated to their reporting or methodological adherence. Overall, the claims on 57 to 93% of the items per publication were confirmed in the item-by-item analysis, with a mean and standard deviation of 81% and 10%, respectively.

CONCLUSION

Only a small proportion of the publications used CLAIM as checklist and supplied filled-out documentation; however, the self-reported checklists may contain errors and should be approached cautiously. We hope that this systematic citation analysis would motivate artificial intelligence community about the importance of proper self-reporting, and encourage researchers, journals, editors, and reviewers to take action to ensure the proper usage of checklists.

CLINICAL RELEVANCE STATEMENT

Only a small percentage of the publications used CLAIM for self-reporting with proof (i.e., filled-out checklist). However, the filled-out checklist proofs may contain errors, e.g., false claims of adherence, and should be approached cautiously. These may indicate inappropriate usage of checklists and necessitate further action by authorities.

KEY POINTS

• Of 118 eligible papers, only 12 (10%) followed the CLAIM checklist for self-reporting with proof (i.e., filled-out checklist). More than half (70; 59%) only mentioned some kind of adherence without providing any proof. • Overall, claims on 57 to 93% of the items were valid in item-by-item confirmation analysis, with a mean and standard deviation of 81% and 10%, respectively. • Even with the checklist proof, the items declared may contain errors and should be approached cautiously.

The endorsement of general and artificial intelligence reporting guidelines in radiological journals: a meta-research study

BACKGROUND

Complete reporting is essential for clinical research. However, the endorsement of reporting guidelines in radiological journals is still unclear. Further, as a field extensively utilizing artificial intelligence (AI), the adoption of both general and AI reporting guidelines would be necessary for enhancing quality and transparency of radiological research. This study aims to investigate the endorsement of general reporting guidelines and those for AI applications in medical imaging in radiological journals, and explore associated journal characteristic variables.

METHODS

This meta-research study screened journals from the Radiology, Nuclear Medicine & Medical Imaging category, Science Citation Index Expanded of the 2022 Journal Citation Reports, and excluded journals not publishing original research, in non-English languages, and instructions for authors unavailable. The endorsement of fifteen general reporting guidelines and ten AI reporting guidelines was rated using a five-level tool: "active strong", "active weak", "passive moderate", "passive weak", and "none". The association between endorsement and journal characteristic variables was evaluated by logistic regression analysis.

RESULTS

We included 117 journals. The top-five endorsed reporting guidelines were CONSORT (Consolidated Standards of Reporting Trials, 58.1%, 68/117), PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses, 54.7%, 64/117), STROBE (STrengthening the Reporting of Observational Studies in Epidemiology, 51.3%, 60/117), STARD (Standards for Reporting of Diagnostic Accuracy, 50.4%, 59/117), and ARRIVE (Animal Research Reporting of In Vivo Experiments, 35.9%, 42/117). The most implemented AI reporting guideline was CLAIM (Checklist for Artificial Intelligence in Medical Imaging, 1.7%, 2/117), while other nine AI reporting guidelines were not mentioned. The Journal Impact Factor quartile and publisher were associated with endorsement of reporting guidelines in radiological journals.

CONCLUSIONS

The general reporting guideline endorsement was suboptimal in radiological journals. The implementation of reporting guidelines for AI applications in medical imaging was extremely low. Their adoption should be strengthened to facilitate quality and transparency of radiological study reporting.

Reporting Quality of Research Studies on AI Applications in Medical Images According to the CLAIM Guidelines in a Radiology Journal With a Strong Prominence in Asia

OBJECTIVE

We aimed to evaluate the reporting quality of research articles that applied deep learning to medical imaging. Using the Checklist for Artificial Intelligence in Medical Imaging (CLAIM) guidelines and a journal with prominence in Asia as a sample, we intended to provide an insight into reporting quality in the Asian region and establish a journal-specific audit.

MATERIALS AND METHODS

A total of 38 articles published in the Korean Journal of Radiology between June 2018 and January 2023 were analyzed. The analysis included calculating the percentage of studies that adhered to each CLAIM item and identifying items that were met by ≤ 50% of the studies. The article review was initially conducted independently by two reviewers, and the consensus results were used for the final analysis. We also compared adherence rates to CLAIM before and after December 2020.

RESULTS

Of the 42 items in the CLAIM guidelines, 12 items (29%) were satisfied by ≤ 50% of the included articles. None of the studies reported handling missing data (item #13). Only one study respectively presented the use of de-identification methods (#12), intended sample size (#19), robustness or sensitivity analysis (#30), and full study protocol (#41). Of the studies, 35% reported the selection of data subsets (#10), 40% reported registration information (#40), and 50% measured inter and intrarater variability (#18). No significant changes were observed in the rates of adherence to these 12 items before and after December 2020.

CONCLUSION

The reporting quality of artificial intelligence studies according to CLAIM guidelines, in our study sample, showed room for improvement. We recommend that the authors and reviewers have a solid understanding of the relevant reporting guidelines and ensure that the essential elements are adequately reported when writing and reviewing the manuscripts for publication.

An overview of meta-analyses on radiomics: more evidence is needed to support clinical translation

OBJECTIVE

To conduct an overview of meta-analyses of radiomics studies assessing their study quality and evidence level.

METHODS

A systematical search was updated via peer-reviewed electronic databases, preprint servers, and systematic review protocol registers until 15 November 2022. Systematic reviews with meta-analysis of primary radiomics studies were included. Their reporting transparency, methodological quality, and risk of bias were assessed by PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses) 2020 checklist, AMSTAR-2 (A MeaSurement Tool to Assess systematic Reviews, version 2) tool, and ROBIS (Risk Of Bias In Systematic reviews) tool, respectively. The evidence level supporting the radiomics for clinical use was rated.

RESULTS

We identified 44 systematic reviews with meta-analyses on radiomics research. The mean ± standard deviation of PRISMA adherence rate was 65 ± 9%. The AMSTAR-2 tool rated 5 and 39 systematic reviews as low and critically low confidence, respectively. The ROBIS assessment resulted low, unclear and high risk in 5, 11, and 28 systematic reviews, respectively. We reperformed 53 meta-analyses in 38 included systematic reviews. There were 3, 7, and 43 meta-analyses rated as convincing, highly suggestive, and weak levels of evidence, respectively. The convincing level of evidence was rated in (1) T2-FLAIR radiomics for IDH-mutant vs IDH-wide type differentiation in low-grade glioma, (2) CT radiomics for COVID-19 vs other viral pneumonia differentiation, and (3) MRI radiomics for high-grade glioma vs brain metastasis differentiation.

CONCLUSIONS

The systematic reviews on radiomics were with suboptimal quality. A limited number of radiomics approaches were supported by convincing level of evidence.

CLINICAL RELEVANCE STATEMENT

The evidence supporting the clinical application of radiomics are insufficient, calling for researches translating radiomics from an academic tool to a practicable adjunct towards clinical deployment.

A systematic review of radiomics in giant cell tumor of bone (GCTB): the potential of analysis on individual radiomics feature for identifying genuine promising imaging biomarkers

PURPOSE

To systematically assess the quality of radiomics research in giant cell tumor of bone (GCTB) and to test the feasibility of analysis at the level of radiomics feature.

METHODS

We searched PubMed, Embase, Web of Science, China National Knowledge Infrastructure, and Wanfang Data to identify articles of GCTB radiomics until 31 July 2022. The studies were assessed by radiomics quality score (RQS), transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD) statement, checklist for artificial intelligence in medical imaging (CLAIM), and modified quality assessment of diagnostic accuracy studies (QUADAS-2) tool. The radiomic features selected for model development were documented.

RESULTS

Nine articles were included. The average of the ideal percentage of RQS, the TRIPOD adherence rate and the CLAIM adherence rate were 26%, 56%, and 57%, respectively. The risk of bias and applicability concerns were mainly related to the index test. The shortness in external validation and open science were repeatedly emphasized. In GCTB radiomics models, the gray level co-occurrence matrix features (40%), first order features (28%), and gray-level run-length matrix features (18%) were most selected features out of all reported features. However, none of the individual feature has appeared repeatably in multiple studies. It is not possible to meta-analyze radiomics features at present.

CONCLUSION

The quality of GCTB radiomics studies is suboptimal. The reporting of individual radiomics feature data is encouraged. The analysis at the level of radiomics feature has potential to generate more practicable evidence for translating radiomics into clinical application.

Prediction of response to preoperative neoadjuvant chemotherapy in extremity high-grade osteosarcoma using X-ray and multiparametric MRI radiomics

PURPOSE

This study aims to evaluate the value of applying X-ray and magnetic resonance imaging (MRI) models based on radiomics feature to predict response of extremity high-grade osteosarcoma to neoadjuvant chemotherapy (NAC).

MATERIALS AND METHODS

A retrospective dataset was assembled involving 102 consecutive patients (training dataset, n = 72; validation dataset, n = 30) diagnosed with extremity high-grade osteosarcoma. The clinical features of age, gender, pathological type, lesion location, bone destruction type, size, alkaline phosphatase (ALP), and lactate dehydrogenase (LDH) were evaluated. Imaging features were extracted from X-ray and multi-parametric MRI (T1-weighted, T2-weighted, and contrast-enhanced T1-weighted) data. Features were selected using a two-stage process comprising minimal-redundancy-maximum-relevance (mRMR) and least absolute shrinkage and selection operator (LASSO) regression. Logistic regression (LR) modelling was then applied to establish models based on clinical, X-ray, and multi-parametric MRI data, as well as combinations of these datasets. Each model was evaluated using sensitivity, specificity, and area under the receiver operating characteristic curve (AUC) with a 95% confidence interval (CI).

RESULTS

AUCs of 5 models using clinical, X-ray radiomics, MRI radiomics, X-ray plus MRI radiomics, and combination of all were 0.760 (95% CI: 0.583-0.937), 0.706 (95% CI: 0.506-0.905), 0.751 (95% CI: 0.572-0.930), 0.796 (95% CI: 0.629-0.963), 0.828 (95% CI: 0.676-0.980), respectively. The DeLong test showed no significant difference between any pair of models (p > 0.05). The combined model yielded higher performance than the clinical and radiomics models as demonstrated by net reclassification improvement (NRI) and integrated difference improvement (IDI) values, respectively. This combined model was also found to be clinically useful in the decision curve analysis (DCA).

CONCLUSION

Modelling based on combination of clinical and radiomics data improves the ability to predict pathological responses to NAC in extremity high-grade osteosarcoma compared to the models based on either clinical or radiomics data.