1
|
Kocak B, Akinci D'Antonoli T, Ates Kus E, Keles A, Kala A, Kose F, Kadioglu M, Solak S, Sunman S, Temiz ZH. Self-reported checklists and quality scoring tools in radiomics: a meta-research. Eur Radiol 2024; 34:5028-5040. [PMID: 38180530 DOI: 10.1007/s00330-023-10487-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/24/2023] [Revised: 11/11/2023] [Accepted: 11/24/2023] [Indexed: 01/06/2024]
Abstract
OBJECTIVE To evaluate the use of reporting checklists and quality scoring tools for self-reporting purposes in radiomics literature. METHODS Literature search was conducted in PubMed (date, April 23, 2023). The radiomics literature was sampled at random after a sample size calculation with a priori power analysis. A systematic assessment for self-reporting, including the use of documentation such as completed checklists or quality scoring tools, was conducted in original research papers. These eligible papers underwent independent evaluation by a panel of nine readers, with three readers assigned to each paper. Automatic annotation was used to assist in this process. Then, a detailed item-by-item confirmation analysis was carried out on papers with checklist documentation, with independent evaluation of two readers. RESULTS The sample size calculation yielded 117 papers. Most of the included papers were retrospective (94%; 110/117), single-center (68%; 80/117), based on their private data (89%; 104/117), and lacked external validation (79%; 93/117). Only seven papers (6%) had at least one self-reported document (Radiomics Quality Score (RQS), Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD), or Checklist for Artificial Intelligence in Medical Imaging (CLAIM)), with a statistically significant binomial test (p < 0.001). Median rate of confirmed items for all three documents was 81% (interquartile range, 6). For quality scoring tools, documented scores were higher than suggested scores, with a mean difference of - 7.2 (standard deviation, 6.8). CONCLUSION Radiomic publications often lack self-reported checklists or quality scoring tools. Even when such documents are provided, it is essential to be cautious, as the accuracy of the reported items or scores may be questionable. CLINICAL RELEVANCE STATEMENT Current state of radiomic literature reveals a notable absence of self-reporting with documentation and inaccurate reporting practices. This critical observation may serve as a catalyst for motivating the radiomics community to adopt and utilize such tools appropriately, thereby fostering rigor, transparency, and reproducibility of their research, moving the field forward. KEY POINTS • In radiomics literature, there has been a notable absence of self-reporting with documentation. • Even if such documents are provided, it is critical to exercise caution because the accuracy of the reported items or scores may be questionable. • Radiomics community needs to be motivated to adopt and appropriately utilize the reporting checklists and quality scoring tools.
Collapse
Affiliation(s)
- Burak Kocak
- Department of Radiology, University of Health Sciences, Basaksehir Cam and Sakura City Hospital, Basaksehir, Istanbul, 34480, Turkey.
| | - Tugba Akinci D'Antonoli
- Institute of Radiology and Nuclear Medicine, Cantonal Hospital Baselland, Liestal, Switzerland
| | - Ece Ates Kus
- Department of Neuroradiology, Klinikum Lippe, Lemgo, Germany
| | - Ali Keles
- Department of Radiology, University of Health Sciences, Basaksehir Cam and Sakura City Hospital, Basaksehir, Istanbul, 34480, Turkey
| | - Ahmet Kala
- Department of Radiology, University of Health Sciences, Basaksehir Cam and Sakura City Hospital, Basaksehir, Istanbul, 34480, Turkey
| | - Fadime Kose
- Department of Radiology, University of Health Sciences, Basaksehir Cam and Sakura City Hospital, Basaksehir, Istanbul, 34480, Turkey
| | - Mehmet Kadioglu
- Department of Radiology, University of Health Sciences, Basaksehir Cam and Sakura City Hospital, Basaksehir, Istanbul, 34480, Turkey
| | - Sila Solak
- Department of Radiology, University of Health Sciences, Basaksehir Cam and Sakura City Hospital, Basaksehir, Istanbul, 34480, Turkey
| | - Seyma Sunman
- Department of Radiology, University of Health Sciences, Basaksehir Cam and Sakura City Hospital, Basaksehir, Istanbul, 34480, Turkey
| | - Zisan Hayriye Temiz
- Department of Radiology, University of Health Sciences, Basaksehir Cam and Sakura City Hospital, Basaksehir, Istanbul, 34480, Turkey
| |
Collapse
|
2
|
Jiang T, Zhao Z, Liu X, Shen C, Mu M, Cai Z, Zhang B. Methodological quality of radiomic-based prognostic studies in gastric cancer: a cross-sectional study. Front Oncol 2023; 13:1161237. [PMID: 37731636 PMCID: PMC10507631 DOI: 10.3389/fonc.2023.1161237] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/08/2023] [Accepted: 08/16/2023] [Indexed: 09/22/2023] Open
Abstract
Background Machine learning radiomics models are increasingly being used to predict gastric cancer prognoses. However, the methodological quality of these models has not been evaluated. Therefore, this study aimed to evaluate the methodological quality of radiomics studies in predicting the prognosis of gastric cancer, summarize their methodological characteristics and performance. Methods The PubMed and Embase databases were searched for radiomics studies used to predict the prognosis of gastric cancer published in last 5 years. The characteristics of the studies and the performance of the models were extracted from the eligible full texts. The methodological quality, reporting completeness and risk of bias of the included studies were evaluated using the RQS, TRIPOD and PROBAST. The discrimination ability scores of the models were also compared. Results Out of 283 identified records, 22 studies met the inclusion criteria. The study endpoints included survival time, treatment response, and recurrence, with reported discriminations ranging between 0.610 and 0.878 in the validation dataset. The mean overall RQS value was 15.32 ± 3.20 (range: 9 to 21). The mean adhered items of the 35 item of TRIPOD checklist was 20.45 ± 1.83. The PROBAST showed all included studies were at high risk of bias. Conclusion The current methodological quality of gastric cancer radiomics studies is insufficient. Large and reasonable sample, prospective, multicenter and rigorously designed studies are required to improve the quality of radiomics models for gastric cancer prediction. Study registration This protocol was prospectively registered in the Open Science Framework Registry (https://osf.io/ja52b).
Collapse
Affiliation(s)
- Tianxiang Jiang
- Department of General Surgery, West China Hospital, Sichuan University, Chengdu, China
- Gastric Cancer Center, West China Hospital, Sichuan University, Chengdu, China
| | - Zhou Zhao
- Department of Gastrointestinal Cancer Center, Chongqing University Cancer Hospital, Chongqing, China
| | - Xueting Liu
- Department of Medical Discipline Construction, West China Hospital, Sichuan University, Chengdu, China
| | - Chaoyong Shen
- Department of General Surgery, West China Hospital, Sichuan University, Chengdu, China
- Gastric Cancer Center, West China Hospital, Sichuan University, Chengdu, China
| | - Mingchun Mu
- Department of General Surgery, West China Hospital, Sichuan University, Chengdu, China
- Gastric Cancer Center, West China Hospital, Sichuan University, Chengdu, China
| | - Zhaolun Cai
- Department of General Surgery, West China Hospital, Sichuan University, Chengdu, China
- Gastric Cancer Center, West China Hospital, Sichuan University, Chengdu, China
| | - Bo Zhang
- Department of General Surgery, West China Hospital, Sichuan University, Chengdu, China
- Gastric Cancer Center, West China Hospital, Sichuan University, Chengdu, China
| |
Collapse
|
3
|
Lin K, Sarnari R, Carr JC, Markl M. Cine MRI-Derived Radiomics Features of the Cardiac Blood Pool: Periodicity, Specificity, and Reproducibility. J Magn Reson Imaging 2023; 58:807-814. [PMID: 36533630 PMCID: PMC10277313 DOI: 10.1002/jmri.28572] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/10/2022] [Revised: 12/05/2022] [Accepted: 12/06/2022] [Indexed: 12/23/2022] Open
Abstract
BACKGROUND Although radiomics features of the left ventricular wall have been used to assess cardiac diseases, radiomics features of the cardiac blood pool have been relatively ignored. PURPOSE To test the hypothesis that cine MRI-derived radiomics features of the cardiac blood pool are associated with cardiac function and motion. STUDY TYPE Retrospective. POPULATION A total of 26 healthy volunteers (51.2 ± 15.6 years; 17 males). FIELD STRENGTH/SEQUENCE A 1.5 T/balanced steady-state free precession (bSSFP). ASSESSMENT The radiomics features (107 features in seven classes) of the blood pool of the left/right ventricle/atrium (LV/RV/LA/RA) were extracted on four-chamber cine images (25 phases). Conventional cardiac function parameters (volumes, ejection fraction [EF] and longitudinal strain) were assessed in each cardiac chamber. Intraobserver- and interobserver agreements of radiomics features of all chambers acquired at all phases were assessed, as well as scan-rescan agreement in a subset of 13 volunteers. STATISTICAL TESTS Pearson correlation coefficients (r) were used to assess the associations between peak values of radiomics features and end-diastolic (or maximal) volume, end-systolic (or minimal) volume, EF, and longitudinal strain of corresponding chambers. Good intraobserver, interobserver, and scan-rescan agreements for radiomics features acquired were defined as intraclass correlation coefficient (ICC) > 0.7 or coefficient of variation (CoV) < 20%. RESULTS Most radiomics features of the blood pool varied periodically throughout the cardiac cycle. Peak values of chamber-specific blood pool radiomics features were correlated with traditional cardiac function and motion indices of corresponding chambers (r: 0.4-0.87). Ninety-three (87%), 86 (80%), and 73 (68%) radiomics features demonstrated good intraobserver, interobserver, and scan-rescan reproducibility, respectively. CONCLUSION Cine MRI-derived radiomics features within LV/RV/LA/RA are associated with traditional cardiac function and motion indices of corresponding chambers and may have the potential to become novel quantitative imaging biomarkers in cardiovascular medicine. EVIDENCE LEVEL 3. TECHNICAL EFFICACY 1.
Collapse
Affiliation(s)
- Kai Lin
- Department of Radiology, Northwestern University, Chicago, Illinois, USA
| | - Roberto Sarnari
- Department of Radiology, Northwestern University, Chicago, Illinois, USA
| | - James C Carr
- Department of Radiology, Northwestern University, Chicago, Illinois, USA
| | - Michael Markl
- Department of Radiology, Northwestern University, Chicago, Illinois, USA
| |
Collapse
|
4
|
Zhao LT, Liu ZY, Xie WF, Shao LZ, Lu J, Tian J, Liu JG. What benefit can be obtained from magnetic resonance imaging diagnosis with artificial intelligence in prostate cancer compared with clinical assessments? Mil Med Res 2023; 10:29. [PMID: 37357263 DOI: 10.1186/s40779-023-00464-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 01/16/2023] [Accepted: 06/07/2023] [Indexed: 06/27/2023] Open
Abstract
The present study aimed to explore the potential of artificial intelligence (AI) methodology based on magnetic resonance (MR) images to aid in the management of prostate cancer (PCa). To this end, we reviewed and summarized the studies comparing the diagnostic and predictive performance for PCa between AI and common clinical assessment methods based on MR images and/or clinical characteristics, thereby investigating whether AI methods are generally superior to common clinical assessment methods for the diagnosis and prediction fields of PCa. First, we found that, in the included studies of the present study, AI methods were generally equal to or better than the clinical assessment methods for the risk assessment of PCa, such as risk stratification of prostate lesions and the prediction of therapeutic outcomes or PCa progression. In particular, for the diagnosis of clinically significant PCa, the AI methods achieved a higher summary receiver operator characteristic curve (SROC-AUC) than that of the clinical assessment methods (0.87 vs. 0.82). For the prediction of adverse pathology, the AI methods also achieved a higher SROC-AUC than that of the clinical assessment methods (0.86 vs. 0.75). Second, as revealed by the radiomics quality score (RQS), the studies included in the present study presented a relatively high total average RQS of 15.2 (11.0-20.0). Further, the scores of the individual RQS elements implied that the AI models in these studies were constructed with relatively perfect and standard radiomics processes, but the exact generalizability and clinical practicality of the AI models should be further validated using higher levels of evidence, such as prospective studies and open-testing datasets.
Collapse
Affiliation(s)
- Li-Tao Zhao
- School of Engineering Medicine, Beihang University, Beijing, 100191, China
- School of Biological Science and Medical Engineering, Beihang University, Beijing, 100191, China
| | - Zhen-Yu Liu
- CAS Key Laboratory of Molecular Imaging, Institute of Automation, Beijing, 100190, China
- University of Chinese Academy of Sciences, Beijing, 100080, China
| | - Wan-Fang Xie
- School of Engineering Medicine, Beihang University, Beijing, 100191, China
- School of Biological Science and Medical Engineering, Beihang University, Beijing, 100191, China
| | - Li-Zhi Shao
- CAS Key Laboratory of Molecular Imaging, Institute of Automation, Beijing, 100190, China
| | - Jian Lu
- Department of Urology, Peking University Third Hospital, Peking University, 100191, Beijing, China.
| | - Jie Tian
- School of Engineering Medicine, Beihang University, Beijing, 100191, China.
- CAS Key Laboratory of Molecular Imaging, Institute of Automation, Beijing, 100190, China.
- Key Laboratory of Big Data-Based Precision Medicine (Beihang University), Ministry of Industry and Information Technology of the People's Republic of China, 100191, Beijing, China.
| | - Jian-Gang Liu
- School of Engineering Medicine, Beihang University, Beijing, 100191, China.
- Key Laboratory of Big Data-Based Precision Medicine (Beihang University), Ministry of Industry and Information Technology of the People's Republic of China, 100191, Beijing, China.
- Beijing Engineering Research Center of Cardiovascular Wisdom Diagnosis and Treatment, Beijing, 100029, China.
| |
Collapse
|
5
|
Cannella R, Vernuccio F, Klontzas ME, Ponsiglione A, Petrash E, Ugga L, Pinto dos Santos D, Cuocolo R. Systematic review with radiomics quality score of cholangiocarcinoma: an EuSoMII Radiomics Auditing Group Initiative. Insights Imaging 2023; 14:21. [PMID: 36720726 PMCID: PMC9889586 DOI: 10.1186/s13244-023-01365-1] [Citation(s) in RCA: 7] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/23/2022] [Accepted: 12/24/2022] [Indexed: 02/02/2023] Open
Abstract
OBJECTIVES To systematically review current research applications of radiomics in patients with cholangiocarcinoma and to assess the quality of CT and MRI radiomics studies. METHODS A systematic search was conducted on PubMed/Medline, Web of Science, and Scopus databases to identify original studies assessing radiomics of cholangiocarcinoma on CT and/or MRI. Three readers with different experience levels independently assessed quality of the studies using the radiomics quality score (RQS). Subgroup analyses were performed according to journal type, year of publication, quartile and impact factor (from the Journal Citation Report database), type of cholangiocarcinoma, imaging modality, and number of patients. RESULTS A total of 38 original studies including 6242 patients (median 134 patients) were selected. The median RQS was 9 (corresponding to 25.0% of the total RQS; IQR 1-13) for reader 1, 8 (22.2%, IQR 3-12) for reader 2, and 10 (27.8%; IQR 5-14) for reader 3. The inter-reader agreement was good with an ICC of 0.75 (95% CI 0.62-0.85) for the total RQS. All studies were retrospective and none of them had phantom assessment, imaging at multiple time points, nor performed cost-effectiveness analysis. The RQS was significantly higher in studies published in journals with impact factor > 4 (median 11 vs. 4, p = 0.048 for reader 1) and including more than 100 patients (median 11.5 vs. 0.5, p < 0.001 for reader 1). CONCLUSIONS Quality of radiomics studies on cholangiocarcinoma is insufficient based on the radiomics quality score. Future research should consider prospective studies with a standardized methodology, validation in multi-institutional external cohorts, and open science data.
Collapse
Affiliation(s)
- Roberto Cannella
- Section of Radiology - Department of Biomedicine, Neuroscience and Advanced Diagnostics (BiND), University Hospital “Paolo Giaccone”, Via del Vespro 129, 90127 Palermo, Italy ,grid.10776.370000 0004 1762 5517Department of Health Promotion, Mother and Child Care, Internal Medicine and Medical Specialties (PROMISE), University of Palermo, Via del Vespro, 129, 90127 Palermo, Italy
| | - Federica Vernuccio
- grid.411474.30000 0004 1760 2630Department of Radiology, University Hospital of Padova, Via Nicolò Giustiniani 2, 35128 Padua, Italy
| | - Michail E. Klontzas
- grid.412481.a0000 0004 0576 5678Department of Medical Imaging, University Hospital of Heraklion, 71110 Voutes, Crete, Greece ,grid.8127.c0000 0004 0576 3437Department of Radiology, School of Medicine, University of Crete, 71003 Heraklion, Crete, Greece ,grid.4834.b0000 0004 0635 685XComputational Biomedicine Laboratory, Institute of Computer Science, Foundation for Research and Technology, Vassilika Vouton, 70013 Crete, Greece
| | - Andrea Ponsiglione
- grid.4691.a0000 0001 0790 385XDepartment of Advanced Biomedical Sciences, University of Naples “Federico II”, Via Sergio Pansini 5, 80131 Naples, Italy
| | - Ekaterina Petrash
- grid.415738.c0000 0000 9216 2496Radiology Department Research Institute of Children’s Oncology and Hematology, FSBI “National Medical Research Center of Oncology n.a. N.N. Blokhin” of Ministry of Health of RF, Kashirskoye Highway 24, Moscow, Russia ,IRA-Labs, Medical Department, Skolkovo, Bolshoi Boulevard, 30, Building 1, Moscow, Russia
| | - Lorenzo Ugga
- grid.4691.a0000 0001 0790 385XDepartment of Advanced Biomedical Sciences, University of Naples “Federico II”, Via Sergio Pansini 5, 80131 Naples, Italy
| | - Daniel Pinto dos Santos
- grid.6190.e0000 0000 8580 3777Department of Diagnostic and Interventional Radiology, Faculty of Medicine and University Hospital Cologne, University of Cologne, Kerpener Str. 62, 50937 Cologne, Germany ,grid.411088.40000 0004 0578 8220Department of Radiology, University Hospital Frankfurt, Theodor-Stern-Kai 7, 60590 Frankfurt, Germany
| | - Renato Cuocolo
- grid.11780.3f0000 0004 1937 0335Department of Medicine, Surgery, and Dentistry, University of Salerno, Via Salvador Allende 43, 84081 Baronissi, SA Italy ,grid.4691.a0000 0001 0790 385XAugmented Reality for Health Monitoring Laboratory (ARHeMLab), Department of Electrical Engineering and Information Technology, University of Naples “Federico II”, Via Sergio Pansini 5, 80131 Naples, Italy
| |
Collapse
|
6
|
Szabo L, Raisi-Estabragh Z, Salih A, McCracken C, Ruiz Pujadas E, Gkontra P, Kiss M, Maurovich-Horvath P, Vago H, Merkely B, Lee AM, Lekadir K, Petersen SE. Clinician's guide to trustworthy and responsible artificial intelligence in cardiovascular imaging. Front Cardiovasc Med 2022; 9:1016032. [PMID: 36426221 PMCID: PMC9681217 DOI: 10.3389/fcvm.2022.1016032] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/10/2022] [Accepted: 10/11/2022] [Indexed: 12/01/2023] Open
Abstract
A growing number of artificial intelligence (AI)-based systems are being proposed and developed in cardiology, driven by the increasing need to deal with the vast amount of clinical and imaging data with the ultimate aim of advancing patient care, diagnosis and prognostication. However, there is a critical gap between the development and clinical deployment of AI tools. A key consideration for implementing AI tools into real-life clinical practice is their "trustworthiness" by end-users. Namely, we must ensure that AI systems can be trusted and adopted by all parties involved, including clinicians and patients. Here we provide a summary of the concepts involved in developing a "trustworthy AI system." We describe the main risks of AI applications and potential mitigation techniques for the wider application of these promising techniques in the context of cardiovascular imaging. Finally, we show why trustworthy AI concepts are important governing forces of AI development.
Collapse
Affiliation(s)
- Liliana Szabo
- William Harvey Research Institute, NIHR Barts Biomedical Research Centre, Queen Mary University of London, London, United Kingdom
- Barts Heart Centre, St Bartholomew's Hospital, Barts Health NHS Trust, London, United Kingdom
- Semmelweis University Heart and Vascular Center, Budapest, Hungary
| | - Zahra Raisi-Estabragh
- William Harvey Research Institute, NIHR Barts Biomedical Research Centre, Queen Mary University of London, London, United Kingdom
- Barts Heart Centre, St Bartholomew's Hospital, Barts Health NHS Trust, London, United Kingdom
| | - Ahmed Salih
- William Harvey Research Institute, NIHR Barts Biomedical Research Centre, Queen Mary University of London, London, United Kingdom
- Barts Heart Centre, St Bartholomew's Hospital, Barts Health NHS Trust, London, United Kingdom
| | - Celeste McCracken
- Division of Cardiovascular Medicine, Radcliffe Department of Medicine, National Institute for Health Research Oxford Biomedical Research Centre, Oxford University Hospitals NHS Foundation Trust, University of Oxford, Oxford, United Kingdom
| | - Esmeralda Ruiz Pujadas
- Departament de Matemàtiques i Informàtica, Artificial Intelligence in Medicine Lab (BCN-AIM), Universitat de Barcelona, Barcelona, Spain
| | - Polyxeni Gkontra
- Departament de Matemàtiques i Informàtica, Artificial Intelligence in Medicine Lab (BCN-AIM), Universitat de Barcelona, Barcelona, Spain
| | - Mate Kiss
- Siemens Healthcare Hungary, Budapest, Hungary
| | - Pal Maurovich-Horvath
- Department of Radiology, Medical Imaging Centre, Semmelweis University, Budapest, Hungary
| | - Hajnalka Vago
- Semmelweis University Heart and Vascular Center, Budapest, Hungary
| | - Bela Merkely
- Semmelweis University Heart and Vascular Center, Budapest, Hungary
| | - Aaron M. Lee
- William Harvey Research Institute, NIHR Barts Biomedical Research Centre, Queen Mary University of London, London, United Kingdom
- Barts Heart Centre, St Bartholomew's Hospital, Barts Health NHS Trust, London, United Kingdom
| | - Karim Lekadir
- Departament de Matemàtiques i Informàtica, Artificial Intelligence in Medicine Lab (BCN-AIM), Universitat de Barcelona, Barcelona, Spain
| | - Steffen E. Petersen
- William Harvey Research Institute, NIHR Barts Biomedical Research Centre, Queen Mary University of London, London, United Kingdom
- Barts Heart Centre, St Bartholomew's Hospital, Barts Health NHS Trust, London, United Kingdom
- Health Data Research UK, London, United Kingdom
- Alan Turing Institute, London, United Kingdom
| |
Collapse
|