Reproducibility of radiomics quality score: an intra- and inter-rater reliability study

OBJECTIVES

To investigate the intra- and inter-rater reliability of the total radiomics quality score (RQS) and the reproducibility of individual RQS items' score in a large multireader study.

METHODS

Nine raters with different backgrounds were randomly assigned to three groups based on their proficiency with RQS utilization: Groups 1 and 2 represented the inter-rater reliability groups with or without prior training in RQS, respectively; group 3 represented the intra-rater reliability group. Thirty-three original research papers on radiomics were evaluated by raters of groups 1 and 2. Of the 33 papers, 17 were evaluated twice with an interval of 1 month by raters of group 3. Intraclass coefficient (ICC) for continuous variables, and Fleiss' and Cohen's kappa (k) statistics for categorical variables were used.

RESULTS

The inter-rater reliability was poor to moderate for total RQS (ICC 0.30-055, p < 0.001) and very low to good for item's reproducibility (k - 0.12 to 0.75) within groups 1 and 2 for both inexperienced and experienced raters. The intra-rater reliability for total RQS was moderate for the less experienced rater (ICC 0.522, p = 0.009), whereas experienced raters showed excellent intra-rater reliability (ICC 0.91-0.99, p < 0.001) between the first and second read. Intra-rater reliability on RQS items' score reproducibility was higher and most of the items had moderate to good intra-rater reliability (k - 0.40 to 1).

CONCLUSIONS

Reproducibility of the total RQS and the score of individual RQS items is low. There is a need for a robust and reproducible assessment method to assess the quality of radiomics research.

CLINICAL RELEVANCE STATEMENT

There is a need for reproducible scoring systems to improve quality of radiomics research and consecutively close the translational gap between research and clinical implementation.

KEY POINTS

• Radiomics quality score has been widely used for the evaluation of radiomics studies. • Although the intra-rater reliability was moderate to excellent, intra- and inter-rater reliability of total score and point-by-point scores were low with radiomics quality score. • A robust, easy-to-use scoring system is needed for the evaluation of radiomics research.

METhodological RadiomICs Score (METRICS): a quality scoring tool for radiomics research endorsed by EuSoMII

PURPOSE

To propose a new quality scoring tool, METhodological RadiomICs Score (METRICS), to assess and improve research quality of radiomics studies.

METHODS

We conducted an online modified Delphi study with a group of international experts. It was performed in three consecutive stages: Stage#1, item preparation; Stage#2, panel discussion among EuSoMII Auditing Group members to identify the items to be voted; and Stage#3, four rounds of the modified Delphi exercise by panelists to determine the items eligible for the METRICS and their weights. The consensus threshold was 75%. Based on the median ranks derived from expert panel opinion and their rank-sum based conversion to importance scores, the category and item weights were calculated.

RESULT

In total, 59 panelists from 19 countries participated in selection and ranking of the items and categories. Final METRICS tool included 30 items within 9 categories. According to their weights, the categories were in descending order of importance: study design, imaging data, image processing and feature extraction, metrics and comparison, testing, feature processing, preparation for modeling, segmentation, and open science. A web application and a repository were developed to streamline the calculation of the METRICS score and to collect feedback from the radiomics community.

CONCLUSION

In this work, we developed a scoring tool for assessing the methodological quality of the radiomics research, with a large international panel and a modified Delphi protocol. With its conditional format to cover methodological variations, it provides a well-constructed framework for the key methodological concepts to assess the quality of radiomic research papers.

CRITICAL RELEVANCE STATEMENT

A quality assessment tool, METhodological RadiomICs Score (METRICS), is made available by a large group of international domain experts, with transparent methodology, aiming at evaluating and improving research quality in radiomics and machine learning.

KEY POINTS

• A methodological scoring tool, METRICS, was developed for assessing the quality of radiomics research, with a large international expert panel and a modified Delphi protocol. • The proposed scoring tool presents expert opinion-based importance weights of categories and items with a transparent methodology for the first time. • METRICS accounts for varying use cases, from handcrafted radiomics to entirely deep learning-based pipelines. • A web application has been developed to help with the calculation of the METRICS score ( https://metricsscore.github.io/metrics/METRICS.html ) and a repository created to collect feedback from the radiomics community ( https://github.com/metricsscore/metrics ).

Delta-radiomics in cancer immunotherapy response prediction: A systematic review

Background

The new immunotherapies have not only changed the oncological therapeutic approach but have also made it necessary to develop new imaging methods for assessing the response to treatment. Delta radiomics consists of the analysis of radiomic features variation between different medical images, usually before and after therapy.

Purpose

This review aims to evaluate the role of delta radiomics in the immunotherapy response assessment.

Methods

A systematic search was performed in PubMed, Scopus, and Web Of Science using "delta radiomics AND immunotherapy" as search terms. The included articles' methodological quality was measured using the Radiomics Quality Score (RQS) tool.

Results

Thirteen articles were finally included in the systematic review. Overall, the RQS of the included studies ranged from 4 to 17, with a mean RQS total of 11,15 ± 4,18 with a corresponding percentage of 30.98 ± 11.61 %. Eleven articles out of 13 performed imaging at multiple time points. All the included articles performed feature reduction. No study carried out prospective validation, decision curve analysis, or cost-effectiveness analysis.

Conclusions

Delta radiomics has been demonstrated useful in evaluating the response in oncologic patients undergoing immunotherapy. The overall quality was found law, due to the lack of prospective design and external validation. Thus, further efforts are needed to bring delta radiomics a step closer to clinical implementation.

A mutation-based radiomics signature predicts response to imatinib in Gastrointestinal Stromal Tumors (GIST)

Objectives

To develop a mutation-based radiomics signature to predict response to imatinib in Gastrointestinal Stromal Tumors (GISTs).

Methods

Eighty-two patients with GIST were enrolled in this retrospective study, including 52 patients from one center that were used to develop the model, and 30 patients from a second center to validate it. Reference standard was the mutational status of tyrosine-protein kinase (KIT) and platelet-derived growth factor α (PDGFRA). Patients were dichotomized in imatinib sensitive (group 0 - mutation in KIT or PDGFRA, different from exon 18-D842V), and imatinib non-responsive (group 1 - PDGFRA exon 18-D842V mutation or absence of mutation in KIT/PDGFRA). Initially, 107 texture features were extracted from the tumor masks of baseline computed tomography scans. Different machine learning methods were then implemented to select the best combination of features for the development of the radiomics signature.

Results

The best performance was obtained with the 5 features selected by the ANOVA model and the Bayes classifier, using a threshold of 0.36. With this setting the radiomics signature had an accuracy and precision for sensitive patients of 82 % (95 % CI:60-95) and 90 % (95 % CI:73-97), respectively. Conversely, a precision of 80 % (95 % CI:34-97) was obtained in non-responsive patients using a threshold of 0.9. Indeed, with the latter setting 4 patients out of 5 were correctly predicted as non-responders.

Conclusions

The results are a first step towards using radiomics to improve the management of patients with GIST, especially when tumor tissue is unavailable for molecular analysis or when molecular profiling is inconclusive.

Looking for appropriateness in follow-up CT of oncologic patients: Results from a cross-sectional study

PURPOSE

The objective of this study was to assess the inappropriateness rate of oncological follow-up CT examinations.

METHODS

Out of 7.000 oncology patients referred for follow-up CT examinations between March and October 2022, a random sample of 10 % was included. Radiology residents assessed the appropriateness using the Italian Society of Medical Oncology (AIOM) guidelines, supervised by senior radiologists. Association between inappropriateness and clinical variables was investigated and variables influencing inappropriateness were analyzed through a binary logistic regression.

RESULTS

Three-hundred-eighty-eight examinations (56.1 %) were consistent with AIOM guidelines. An additional 100 (14.5  %) examinations did not follow the recommended schedule but were nevertheless considered appropriate because of suspected recurrence/progression (10.7 %) or adverse event requiring imaging assessment (3.8 %). Two-hundred-four (29.4 %) examinations were rated as inappropriate. Inappropriateness causes were as follows: CT not included in the relevant guideline (n = 47); CT extended to additional anatomical regions (n = 59); CT requested at a shorter time-interval (n = 98). No statistically significant difference was found in age, sex, scan region, and primary cancer between appropriate and inappropriate examinations. The only variable significantly associated with inappropriateness was being referred by a specific hospital unit named "unit 2" in the study (p = 0.009), which was demonstrated to be the only appropriateness independent predictor (OR 1.952).

CONCLUSION

This study shows that majority of oncological patients referred for follow-up CT follows standard guidelines. However, a non-negligible proportion was rated as inappropriate, mainly due to the shorter time-interval. No clinical variable was associated with inappropriateness, except for referral by a specific hospital unit.

Radiomics Applications in Spleen Imaging: A Systematic Review and Methodological Quality Assessment

The spleen, often referred to as the "forgotten organ", plays numerous important roles in various diseases. Recently, there has been an increased interest in the application of radiomics in different areas of medical imaging. This systematic review aims to assess the current state of the art and evaluate the methodological quality of radiomics applications in spleen imaging. A systematic search was conducted on PubMed, Scopus, and Web of Science. All the studies were analyzed, and several characteristics, such as year of publication, research objectives, and number of patients, were collected. The methodological quality was evaluated using the radiomics quality score (RQS). Fourteen articles were ultimately included in this review. The majority of these articles were published in non-radiological journals (78%), utilized computed tomography (CT) for extracting radiomic features (71%), and involved not only the spleen but also other organs for feature extraction (71%). Overall, the included papers achieved an average RQS total score of 9.71 ± 6.37, corresponding to an RQS percentage of 27.77 ± 16.04. In conclusion, radiomics applications in spleen imaging demonstrate promising results in various clinical scenarios. However, despite all the included papers reporting positive outcomes, there is a lack of consistency in the methodological approaches employed.

A first look into radiomics application in testicular imaging: A systematic review

The aim of this systematic review was to evaluate the state of the art of radiomics in testicular imaging by assessing the quality of radiomic workflow using the Radiomics Quality Score (RQS) and the Quality Assessment of Diagnostic Accuracy Studies-2 (QUADAS-2). A systematic literature search was performed to find potentially relevant articles on the applications of radiomics in testicular imaging, and 6 final articles were extracted. The mean RQS was 11,33 ± 3,88 resulting in a percentage of 31,48% ± 10,78%. Regarding QUADAS-2 criteria, no relevant biases were found in the included papers in the patient selection, index test, reference standard criteria and flow-and-timing domain. In conclusion, despite the publication of promising studies, radiomic research on testicular imaging is in its very beginning and still hindered by methodological limitations, and the potential applications of radiomics for this field are still largely unexplored.

Diagnostic and Therapeutic Radiology of the GI Tract, Liver, and Pancreas in Patients with COVID

Coronavirus disease 2019 (COVID-19) pulmonary involvement has been extensively reported in the literature. Current data highlight how COVID-19 is a systemic disease, affecting many other organs, including the gastrointestinal, hepatobiliary, and pancreatic organs. Recently, these organs have been investigated using imaging modalities of ultrasound and particularly computed tomography. Radiological findings of the gastrointestinal, hepatic, and pancreatic involvement in patients with COVID-19 are generally nonspecific but are nonetheless helpful to evaluate and manage COVID-19 patients with involvement of these organs.