1
|
Tejani AS, Klontzas ME, Gatti AA, Mongan JT, Moy L, Park SH, Kahn CE. Checklist for Artificial Intelligence in Medical Imaging (CLAIM): 2024 Update. Radiol Artif Intell 2024; 6:e240300. [PMID: 38809149 DOI: 10.1148/ryai.240300] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 05/30/2024]
Affiliation(s)
- Ali S Tejani
- From the Department of Radiology, University of Texas Southwestern Medical Center, Dallas, Tex (A.S.T.); Department of Radiology, University of Crete School of Medicine, Heraklion, Crete, Greece (M.E.K.); Department of Medical Imaging, University Hospital of Heraklion, Heraklion, Crete, Greece (M.E.K.); Department of Radiology, Stanford University, Stanford, Calif (A.A.G.); Department of Radiology and Biomedical Imaging, University of California San Francisco, San Francisco, Calif (J.T.M.); Department of Radiology, New York University Grossman School of Medicine, New York, NY (L.M.); Department of Radiology and Research Institute of Radiology, Asan Medical Center, University of Ulsan College of Medicine, Seoul, South Korea (S.H.P.); and Department of Radiology and Institute for Biomedical Informatics, University of Pennsylvania, 3400 Spruce St, Philadelphia, PA 19104-6243 (C.E.K.)
| | - Michail E Klontzas
- From the Department of Radiology, University of Texas Southwestern Medical Center, Dallas, Tex (A.S.T.); Department of Radiology, University of Crete School of Medicine, Heraklion, Crete, Greece (M.E.K.); Department of Medical Imaging, University Hospital of Heraklion, Heraklion, Crete, Greece (M.E.K.); Department of Radiology, Stanford University, Stanford, Calif (A.A.G.); Department of Radiology and Biomedical Imaging, University of California San Francisco, San Francisco, Calif (J.T.M.); Department of Radiology, New York University Grossman School of Medicine, New York, NY (L.M.); Department of Radiology and Research Institute of Radiology, Asan Medical Center, University of Ulsan College of Medicine, Seoul, South Korea (S.H.P.); and Department of Radiology and Institute for Biomedical Informatics, University of Pennsylvania, 3400 Spruce St, Philadelphia, PA 19104-6243 (C.E.K.)
| | - Anthony A Gatti
- From the Department of Radiology, University of Texas Southwestern Medical Center, Dallas, Tex (A.S.T.); Department of Radiology, University of Crete School of Medicine, Heraklion, Crete, Greece (M.E.K.); Department of Medical Imaging, University Hospital of Heraklion, Heraklion, Crete, Greece (M.E.K.); Department of Radiology, Stanford University, Stanford, Calif (A.A.G.); Department of Radiology and Biomedical Imaging, University of California San Francisco, San Francisco, Calif (J.T.M.); Department of Radiology, New York University Grossman School of Medicine, New York, NY (L.M.); Department of Radiology and Research Institute of Radiology, Asan Medical Center, University of Ulsan College of Medicine, Seoul, South Korea (S.H.P.); and Department of Radiology and Institute for Biomedical Informatics, University of Pennsylvania, 3400 Spruce St, Philadelphia, PA 19104-6243 (C.E.K.)
| | - John T Mongan
- From the Department of Radiology, University of Texas Southwestern Medical Center, Dallas, Tex (A.S.T.); Department of Radiology, University of Crete School of Medicine, Heraklion, Crete, Greece (M.E.K.); Department of Medical Imaging, University Hospital of Heraklion, Heraklion, Crete, Greece (M.E.K.); Department of Radiology, Stanford University, Stanford, Calif (A.A.G.); Department of Radiology and Biomedical Imaging, University of California San Francisco, San Francisco, Calif (J.T.M.); Department of Radiology, New York University Grossman School of Medicine, New York, NY (L.M.); Department of Radiology and Research Institute of Radiology, Asan Medical Center, University of Ulsan College of Medicine, Seoul, South Korea (S.H.P.); and Department of Radiology and Institute for Biomedical Informatics, University of Pennsylvania, 3400 Spruce St, Philadelphia, PA 19104-6243 (C.E.K.)
| | - Linda Moy
- From the Department of Radiology, University of Texas Southwestern Medical Center, Dallas, Tex (A.S.T.); Department of Radiology, University of Crete School of Medicine, Heraklion, Crete, Greece (M.E.K.); Department of Medical Imaging, University Hospital of Heraklion, Heraklion, Crete, Greece (M.E.K.); Department of Radiology, Stanford University, Stanford, Calif (A.A.G.); Department of Radiology and Biomedical Imaging, University of California San Francisco, San Francisco, Calif (J.T.M.); Department of Radiology, New York University Grossman School of Medicine, New York, NY (L.M.); Department of Radiology and Research Institute of Radiology, Asan Medical Center, University of Ulsan College of Medicine, Seoul, South Korea (S.H.P.); and Department of Radiology and Institute for Biomedical Informatics, University of Pennsylvania, 3400 Spruce St, Philadelphia, PA 19104-6243 (C.E.K.)
| | - Seong Ho Park
- From the Department of Radiology, University of Texas Southwestern Medical Center, Dallas, Tex (A.S.T.); Department of Radiology, University of Crete School of Medicine, Heraklion, Crete, Greece (M.E.K.); Department of Medical Imaging, University Hospital of Heraklion, Heraklion, Crete, Greece (M.E.K.); Department of Radiology, Stanford University, Stanford, Calif (A.A.G.); Department of Radiology and Biomedical Imaging, University of California San Francisco, San Francisco, Calif (J.T.M.); Department of Radiology, New York University Grossman School of Medicine, New York, NY (L.M.); Department of Radiology and Research Institute of Radiology, Asan Medical Center, University of Ulsan College of Medicine, Seoul, South Korea (S.H.P.); and Department of Radiology and Institute for Biomedical Informatics, University of Pennsylvania, 3400 Spruce St, Philadelphia, PA 19104-6243 (C.E.K.)
| | - Charles E Kahn
- From the Department of Radiology, University of Texas Southwestern Medical Center, Dallas, Tex (A.S.T.); Department of Radiology, University of Crete School of Medicine, Heraklion, Crete, Greece (M.E.K.); Department of Medical Imaging, University Hospital of Heraklion, Heraklion, Crete, Greece (M.E.K.); Department of Radiology, Stanford University, Stanford, Calif (A.A.G.); Department of Radiology and Biomedical Imaging, University of California San Francisco, San Francisco, Calif (J.T.M.); Department of Radiology, New York University Grossman School of Medicine, New York, NY (L.M.); Department of Radiology and Research Institute of Radiology, Asan Medical Center, University of Ulsan College of Medicine, Seoul, South Korea (S.H.P.); and Department of Radiology and Institute for Biomedical Informatics, University of Pennsylvania, 3400 Spruce St, Philadelphia, PA 19104-6243 (C.E.K.)
| |
Collapse
|
2
|
Walston SL, Tatekawa H, Takita H, Miki Y, Ueda D. Evaluating Biases and Quality Issues in Intermodality Image Translation Studies for Neuroradiology: A Systematic Review. AJNR Am J Neuroradiol 2024; 45:826-832. [PMID: 38663993 PMCID: PMC11288590 DOI: 10.3174/ajnr.a8211] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/21/2023] [Accepted: 01/27/2024] [Indexed: 06/09/2024]
Abstract
BACKGROUND Intermodality image-to-image translation is an artificial intelligence technique for generating one technique from another. PURPOSE This review was designed to systematically identify and quantify biases and quality issues preventing validation and clinical application of artificial intelligence models for intermodality image-to-image translation of brain imaging. DATA SOURCES PubMed, Scopus, and IEEE Xplore were searched through August 2, 2023, for artificial intelligence-based image translation models of radiologic brain images. STUDY SELECTION This review collected 102 works published between April 2017 and August 2023. DATA ANALYSIS Eligible studies were evaluated for quality using the Checklist for Artificial Intelligence in Medical Imaging (CLAIM) and for bias using the Prediction model Risk Of Bias ASsessment Tool (PROBAST). Medically-focused article adherence was compared with that of engineering-focused articles overall with the Mann-Whitney U test and for each criterion using the Fisher exact test. DATA SYNTHESIS Median adherence to the relevant CLAIM criteria was 69% and 38% for PROBAST questions. CLAIM adherence was lower for engineering-focused articles compared with medically-focused articles (65% versus 73%, P < .001). Engineering-focused studies had higher adherence for model description criteria, and medically-focused studies had higher adherence for data set and evaluation descriptions. LIMITATIONS Our review is limited by the study design and model heterogeneity. CONCLUSIONS Nearly all studies revealed critical issues preventing clinical application, with engineering-focused studies showing higher adherence for the technical model description but significantly lower overall adherence than medically-focused studies. The pursuit of clinical application requires collaboration from both fields to improve reporting.
Collapse
Affiliation(s)
- Shannon L Walston
- From the Department of Diagnostic and Interventional Radiology (S.L.W., H.Tatekawa, H.Takita, Y.M., D.U.), Graduate School of Medicine, Osaka Metropolitan University, Osaka, Japan
| | - Hiroyuki Tatekawa
- From the Department of Diagnostic and Interventional Radiology (S.L.W., H.Tatekawa, H.Takita, Y.M., D.U.), Graduate School of Medicine, Osaka Metropolitan University, Osaka, Japan
| | - Hirotaka Takita
- From the Department of Diagnostic and Interventional Radiology (S.L.W., H.Tatekawa, H.Takita, Y.M., D.U.), Graduate School of Medicine, Osaka Metropolitan University, Osaka, Japan
| | - Yukio Miki
- From the Department of Diagnostic and Interventional Radiology (S.L.W., H.Tatekawa, H.Takita, Y.M., D.U.), Graduate School of Medicine, Osaka Metropolitan University, Osaka, Japan
| | - Daiju Ueda
- From the Department of Diagnostic and Interventional Radiology (S.L.W., H.Tatekawa, H.Takita, Y.M., D.U.), Graduate School of Medicine, Osaka Metropolitan University, Osaka, Japan
- Smart Life Science Lab (D.U.), Center for Health Science Innovation, Osaka Metropolitan University, Osaka, Japan
| |
Collapse
|
3
|
Zhong J, Xing Y, Lu J, Zhang G, Mao S, Chen H, Yin Q, Cen Q, Jiang R, Hu Y, Ding D, Ge X, Zhang H, Yao W. The endorsement of general and artificial intelligence reporting guidelines in radiological journals: a meta-research study. BMC Med Res Methodol 2023; 23:292. [PMID: 38093215 PMCID: PMC10717715 DOI: 10.1186/s12874-023-02117-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/28/2023] [Accepted: 12/01/2023] [Indexed: 12/17/2023] Open
Abstract
BACKGROUND Complete reporting is essential for clinical research. However, the endorsement of reporting guidelines in radiological journals is still unclear. Further, as a field extensively utilizing artificial intelligence (AI), the adoption of both general and AI reporting guidelines would be necessary for enhancing quality and transparency of radiological research. This study aims to investigate the endorsement of general reporting guidelines and those for AI applications in medical imaging in radiological journals, and explore associated journal characteristic variables. METHODS This meta-research study screened journals from the Radiology, Nuclear Medicine & Medical Imaging category, Science Citation Index Expanded of the 2022 Journal Citation Reports, and excluded journals not publishing original research, in non-English languages, and instructions for authors unavailable. The endorsement of fifteen general reporting guidelines and ten AI reporting guidelines was rated using a five-level tool: "active strong", "active weak", "passive moderate", "passive weak", and "none". The association between endorsement and journal characteristic variables was evaluated by logistic regression analysis. RESULTS We included 117 journals. The top-five endorsed reporting guidelines were CONSORT (Consolidated Standards of Reporting Trials, 58.1%, 68/117), PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses, 54.7%, 64/117), STROBE (STrengthening the Reporting of Observational Studies in Epidemiology, 51.3%, 60/117), STARD (Standards for Reporting of Diagnostic Accuracy, 50.4%, 59/117), and ARRIVE (Animal Research Reporting of In Vivo Experiments, 35.9%, 42/117). The most implemented AI reporting guideline was CLAIM (Checklist for Artificial Intelligence in Medical Imaging, 1.7%, 2/117), while other nine AI reporting guidelines were not mentioned. The Journal Impact Factor quartile and publisher were associated with endorsement of reporting guidelines in radiological journals. CONCLUSIONS The general reporting guideline endorsement was suboptimal in radiological journals. The implementation of reporting guidelines for AI applications in medical imaging was extremely low. Their adoption should be strengthened to facilitate quality and transparency of radiological study reporting.
Collapse
Affiliation(s)
- Jingyu Zhong
- Department of Imaging, Tongren Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, 200336, China
| | - Yue Xing
- Department of Imaging, Tongren Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, 200336, China
| | - Junjie Lu
- Department of Epidemiology and Population Health, Stanford University School of Medicine, Stanford, CA, 94305, USA
| | - Guangcheng Zhang
- Department of Orthopedics, Shanghai Sixth People's Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, 200233, China
| | - Shiqi Mao
- Department of Medical Oncology, Shanghai Pulmonary Hospital, Tongji University School of Medicine, Shanghai, 200433, China
| | - Haoda Chen
- Department of General Surgery, Pancreatic Disease Center, Ruijin Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, 200025, China
| | - Qian Yin
- Department of Pathology, Shanghai Sixth People's Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, 200233, China
| | - Qingqing Cen
- Department of Dermatology, Shanghai Ninth People's Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, 200011, China
| | - Run Jiang
- Department of Pharmacovigilance, Shanghai Hansoh BioMedical Co., Ltd., Shanghai, 201203, China
| | - Yangfan Hu
- Department of Imaging, Tongren Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, 200336, China
| | - Defang Ding
- Department of Imaging, Tongren Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, 200336, China
| | - Xiang Ge
- Department of Imaging, Tongren Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, 200336, China
| | - Huan Zhang
- Department of Radiology, Ruijin Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, 200025, China.
| | - Weiwu Yao
- Department of Imaging, Tongren Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, 200336, China.
| |
Collapse
|
4
|
Kocak B, Bulut E, Bayrak ON, Okumus AA, Altun O, Borekci Arvas Z, Kavukoglu I. NEgatiVE results in Radiomics research (NEVER): A meta-research study of publication bias in leading radiology journals. Eur J Radiol 2023; 163:110830. [PMID: 37119709 DOI: 10.1016/j.ejrad.2023.110830] [Citation(s) in RCA: 8] [Impact Index Per Article: 8.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/07/2023] [Revised: 04/03/2023] [Accepted: 04/05/2023] [Indexed: 05/01/2023]
Abstract
PURPOSE The purpose of this study was to conduct a meta-research of radiomics-related articles for the publication of negative results, with a focus on the leading clinical radiology journals due to their purportedly high editorial standards. METHODS A literature search was performed in PubMed to identify original research studies on radiomics (last search date: August 16th, 2022). The search was restricted to studies published in Q1 clinical radiology journals indexed by Scopus and Web of Science. Following an a priori power analysis based on our null hypothesis, a random sampling of the published literature was conducted. Besides the six baseline study characteristics, a total of three items about publication bias were evaluated. Agreement between raters was analyzed. Disagreements were resolved through consensus. Statistical synthesis of the qualitative evaluations was presented. RESULTS Following a priori power analysis, we included a random sample of 149 publications in this study. Most of the publications were retrospective (95%; 142/149), based on private data (91%; 136/149), centered on a single institution (75%; 111/149), and lacked external validation (81%; 121/149). Slightly fewer than half (44%; 66/149) made no comparison to non-radiomic approaches. Overall, only one study (1%; 1/149) reported negative results for radiomics, yielding a statistically significant binomial test (p < 0.0001). CONCLUSION The top clinical radiology journals almost never publish negative results, having a strong bias toward publishing positive results. Almost half of the publications did not even compare their approach with a non-radiomic method.
Collapse
Affiliation(s)
- Burak Kocak
- Department of Radiology, University of Health Sciences, Basaksehir Cam and Sakura City Hospital, Istanbul, Turkey.
| | - Elif Bulut
- Department of Radiology, University of Health Sciences, Basaksehir Cam and Sakura City Hospital, Istanbul, Turkey
| | - Osman Nuri Bayrak
- Department of Radiology, University of Health Sciences, Basaksehir Cam and Sakura City Hospital, Istanbul, Turkey
| | - Ahmet Arda Okumus
- Department of Radiology, University of Health Sciences, Basaksehir Cam and Sakura City Hospital, Istanbul, Turkey
| | - Omer Altun
- Department of Radiology, University of Health Sciences, Basaksehir Cam and Sakura City Hospital, Istanbul, Turkey
| | - Zeynep Borekci Arvas
- Department of Radiology, University of Health Sciences, Basaksehir Cam and Sakura City Hospital, Istanbul, Turkey
| | - Irem Kavukoglu
- Department of Radiology, University of Health Sciences, Basaksehir Cam and Sakura City Hospital, Istanbul, Turkey
| |
Collapse
|