1
|
Kocak B, Akinci D'Antonoli T, Ates Kus E, Keles A, Kala A, Kose F, Kadioglu M, Solak S, Sunman S, Temiz ZH. Self-reported checklists and quality scoring tools in radiomics: a meta-research. Eur Radiol 2024; 34:5028-5040. [PMID: 38180530 DOI: 10.1007/s00330-023-10487-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/24/2023] [Revised: 11/11/2023] [Accepted: 11/24/2023] [Indexed: 01/06/2024]
Abstract
OBJECTIVE To evaluate the use of reporting checklists and quality scoring tools for self-reporting purposes in radiomics literature. METHODS Literature search was conducted in PubMed (date, April 23, 2023). The radiomics literature was sampled at random after a sample size calculation with a priori power analysis. A systematic assessment for self-reporting, including the use of documentation such as completed checklists or quality scoring tools, was conducted in original research papers. These eligible papers underwent independent evaluation by a panel of nine readers, with three readers assigned to each paper. Automatic annotation was used to assist in this process. Then, a detailed item-by-item confirmation analysis was carried out on papers with checklist documentation, with independent evaluation of two readers. RESULTS The sample size calculation yielded 117 papers. Most of the included papers were retrospective (94%; 110/117), single-center (68%; 80/117), based on their private data (89%; 104/117), and lacked external validation (79%; 93/117). Only seven papers (6%) had at least one self-reported document (Radiomics Quality Score (RQS), Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD), or Checklist for Artificial Intelligence in Medical Imaging (CLAIM)), with a statistically significant binomial test (p < 0.001). Median rate of confirmed items for all three documents was 81% (interquartile range, 6). For quality scoring tools, documented scores were higher than suggested scores, with a mean difference of - 7.2 (standard deviation, 6.8). CONCLUSION Radiomic publications often lack self-reported checklists or quality scoring tools. Even when such documents are provided, it is essential to be cautious, as the accuracy of the reported items or scores may be questionable. CLINICAL RELEVANCE STATEMENT Current state of radiomic literature reveals a notable absence of self-reporting with documentation and inaccurate reporting practices. This critical observation may serve as a catalyst for motivating the radiomics community to adopt and utilize such tools appropriately, thereby fostering rigor, transparency, and reproducibility of their research, moving the field forward. KEY POINTS • In radiomics literature, there has been a notable absence of self-reporting with documentation. • Even if such documents are provided, it is critical to exercise caution because the accuracy of the reported items or scores may be questionable. • Radiomics community needs to be motivated to adopt and appropriately utilize the reporting checklists and quality scoring tools.
Collapse
Affiliation(s)
- Burak Kocak
- Department of Radiology, University of Health Sciences, Basaksehir Cam and Sakura City Hospital, Basaksehir, Istanbul, 34480, Turkey.
| | - Tugba Akinci D'Antonoli
- Institute of Radiology and Nuclear Medicine, Cantonal Hospital Baselland, Liestal, Switzerland
| | - Ece Ates Kus
- Department of Neuroradiology, Klinikum Lippe, Lemgo, Germany
| | - Ali Keles
- Department of Radiology, University of Health Sciences, Basaksehir Cam and Sakura City Hospital, Basaksehir, Istanbul, 34480, Turkey
| | - Ahmet Kala
- Department of Radiology, University of Health Sciences, Basaksehir Cam and Sakura City Hospital, Basaksehir, Istanbul, 34480, Turkey
| | - Fadime Kose
- Department of Radiology, University of Health Sciences, Basaksehir Cam and Sakura City Hospital, Basaksehir, Istanbul, 34480, Turkey
| | - Mehmet Kadioglu
- Department of Radiology, University of Health Sciences, Basaksehir Cam and Sakura City Hospital, Basaksehir, Istanbul, 34480, Turkey
| | - Sila Solak
- Department of Radiology, University of Health Sciences, Basaksehir Cam and Sakura City Hospital, Basaksehir, Istanbul, 34480, Turkey
| | - Seyma Sunman
- Department of Radiology, University of Health Sciences, Basaksehir Cam and Sakura City Hospital, Basaksehir, Istanbul, 34480, Turkey
| | - Zisan Hayriye Temiz
- Department of Radiology, University of Health Sciences, Basaksehir Cam and Sakura City Hospital, Basaksehir, Istanbul, 34480, Turkey
| |
Collapse
|
2
|
Tejani AS, Klontzas ME, Gatti AA, Mongan JT, Moy L, Park SH, Kahn CE. Checklist for Artificial Intelligence in Medical Imaging (CLAIM): 2024 Update. Radiol Artif Intell 2024; 6:e240300. [PMID: 38809149 DOI: 10.1148/ryai.240300] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 05/30/2024]
Affiliation(s)
- Ali S Tejani
- From the Department of Radiology, University of Texas Southwestern Medical Center, Dallas, Tex (A.S.T.); Department of Radiology, University of Crete School of Medicine, Heraklion, Crete, Greece (M.E.K.); Department of Medical Imaging, University Hospital of Heraklion, Heraklion, Crete, Greece (M.E.K.); Department of Radiology, Stanford University, Stanford, Calif (A.A.G.); Department of Radiology and Biomedical Imaging, University of California San Francisco, San Francisco, Calif (J.T.M.); Department of Radiology, New York University Grossman School of Medicine, New York, NY (L.M.); Department of Radiology and Research Institute of Radiology, Asan Medical Center, University of Ulsan College of Medicine, Seoul, South Korea (S.H.P.); and Department of Radiology and Institute for Biomedical Informatics, University of Pennsylvania, 3400 Spruce St, Philadelphia, PA 19104-6243 (C.E.K.)
| | - Michail E Klontzas
- From the Department of Radiology, University of Texas Southwestern Medical Center, Dallas, Tex (A.S.T.); Department of Radiology, University of Crete School of Medicine, Heraklion, Crete, Greece (M.E.K.); Department of Medical Imaging, University Hospital of Heraklion, Heraklion, Crete, Greece (M.E.K.); Department of Radiology, Stanford University, Stanford, Calif (A.A.G.); Department of Radiology and Biomedical Imaging, University of California San Francisco, San Francisco, Calif (J.T.M.); Department of Radiology, New York University Grossman School of Medicine, New York, NY (L.M.); Department of Radiology and Research Institute of Radiology, Asan Medical Center, University of Ulsan College of Medicine, Seoul, South Korea (S.H.P.); and Department of Radiology and Institute for Biomedical Informatics, University of Pennsylvania, 3400 Spruce St, Philadelphia, PA 19104-6243 (C.E.K.)
| | - Anthony A Gatti
- From the Department of Radiology, University of Texas Southwestern Medical Center, Dallas, Tex (A.S.T.); Department of Radiology, University of Crete School of Medicine, Heraklion, Crete, Greece (M.E.K.); Department of Medical Imaging, University Hospital of Heraklion, Heraklion, Crete, Greece (M.E.K.); Department of Radiology, Stanford University, Stanford, Calif (A.A.G.); Department of Radiology and Biomedical Imaging, University of California San Francisco, San Francisco, Calif (J.T.M.); Department of Radiology, New York University Grossman School of Medicine, New York, NY (L.M.); Department of Radiology and Research Institute of Radiology, Asan Medical Center, University of Ulsan College of Medicine, Seoul, South Korea (S.H.P.); and Department of Radiology and Institute for Biomedical Informatics, University of Pennsylvania, 3400 Spruce St, Philadelphia, PA 19104-6243 (C.E.K.)
| | - John T Mongan
- From the Department of Radiology, University of Texas Southwestern Medical Center, Dallas, Tex (A.S.T.); Department of Radiology, University of Crete School of Medicine, Heraklion, Crete, Greece (M.E.K.); Department of Medical Imaging, University Hospital of Heraklion, Heraklion, Crete, Greece (M.E.K.); Department of Radiology, Stanford University, Stanford, Calif (A.A.G.); Department of Radiology and Biomedical Imaging, University of California San Francisco, San Francisco, Calif (J.T.M.); Department of Radiology, New York University Grossman School of Medicine, New York, NY (L.M.); Department of Radiology and Research Institute of Radiology, Asan Medical Center, University of Ulsan College of Medicine, Seoul, South Korea (S.H.P.); and Department of Radiology and Institute for Biomedical Informatics, University of Pennsylvania, 3400 Spruce St, Philadelphia, PA 19104-6243 (C.E.K.)
| | - Linda Moy
- From the Department of Radiology, University of Texas Southwestern Medical Center, Dallas, Tex (A.S.T.); Department of Radiology, University of Crete School of Medicine, Heraklion, Crete, Greece (M.E.K.); Department of Medical Imaging, University Hospital of Heraklion, Heraklion, Crete, Greece (M.E.K.); Department of Radiology, Stanford University, Stanford, Calif (A.A.G.); Department of Radiology and Biomedical Imaging, University of California San Francisco, San Francisco, Calif (J.T.M.); Department of Radiology, New York University Grossman School of Medicine, New York, NY (L.M.); Department of Radiology and Research Institute of Radiology, Asan Medical Center, University of Ulsan College of Medicine, Seoul, South Korea (S.H.P.); and Department of Radiology and Institute for Biomedical Informatics, University of Pennsylvania, 3400 Spruce St, Philadelphia, PA 19104-6243 (C.E.K.)
| | - Seong Ho Park
- From the Department of Radiology, University of Texas Southwestern Medical Center, Dallas, Tex (A.S.T.); Department of Radiology, University of Crete School of Medicine, Heraklion, Crete, Greece (M.E.K.); Department of Medical Imaging, University Hospital of Heraklion, Heraklion, Crete, Greece (M.E.K.); Department of Radiology, Stanford University, Stanford, Calif (A.A.G.); Department of Radiology and Biomedical Imaging, University of California San Francisco, San Francisco, Calif (J.T.M.); Department of Radiology, New York University Grossman School of Medicine, New York, NY (L.M.); Department of Radiology and Research Institute of Radiology, Asan Medical Center, University of Ulsan College of Medicine, Seoul, South Korea (S.H.P.); and Department of Radiology and Institute for Biomedical Informatics, University of Pennsylvania, 3400 Spruce St, Philadelphia, PA 19104-6243 (C.E.K.)
| | - Charles E Kahn
- From the Department of Radiology, University of Texas Southwestern Medical Center, Dallas, Tex (A.S.T.); Department of Radiology, University of Crete School of Medicine, Heraklion, Crete, Greece (M.E.K.); Department of Medical Imaging, University Hospital of Heraklion, Heraklion, Crete, Greece (M.E.K.); Department of Radiology, Stanford University, Stanford, Calif (A.A.G.); Department of Radiology and Biomedical Imaging, University of California San Francisco, San Francisco, Calif (J.T.M.); Department of Radiology, New York University Grossman School of Medicine, New York, NY (L.M.); Department of Radiology and Research Institute of Radiology, Asan Medical Center, University of Ulsan College of Medicine, Seoul, South Korea (S.H.P.); and Department of Radiology and Institute for Biomedical Informatics, University of Pennsylvania, 3400 Spruce St, Philadelphia, PA 19104-6243 (C.E.K.)
| |
Collapse
|
3
|
Kocak B, Borgheresi A, Ponsiglione A, Andreychenko AE, Cavallo AU, Stanzione A, Doniselli FM, Vernuccio F, Triantafyllou M, Cannella R, Trotta R, Ghezzo S, Akinci D'Antonoli T, Cuocolo R. Explanation and Elaboration with Examples for CLEAR (CLEAR-E3): an EuSoMII Radiomics Auditing Group Initiative. Eur Radiol Exp 2024; 8:72. [PMID: 38740707 DOI: 10.1186/s41747-024-00471-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/18/2024] [Accepted: 04/17/2024] [Indexed: 05/16/2024] Open
Abstract
Overall quality of radiomics research has been reported as low in literature, which constitutes a major challenge to improve. Consistent, transparent, and accurate reporting is critical, which can be accomplished with systematic use of reporting guidelines. The CheckList for EvaluAtion of Radiomics research (CLEAR) was previously developed to assist authors in reporting their radiomic research and to assist reviewers in their evaluation. To take full advantage of CLEAR, further explanation and elaboration of each item, as well as literature examples, may be useful. The main goal of this work, Explanation and Elaboration with Examples for CLEAR (CLEAR-E3), is to improve CLEAR's usability and dissemination. In this international collaborative effort, members of the European Society of Medical Imaging Informatics-Radiomics Auditing Group searched radiomics literature to identify representative reporting examples for each CLEAR item. At least two examples, demonstrating optimal reporting, were presented for each item. All examples were selected from open-access articles, allowing users to easily consult the corresponding full-text articles. In addition to these, each CLEAR item's explanation was further expanded and elaborated. For easier access, the resulting document is available at https://radiomic.github.io/CLEAR-E3/ . As a complementary effort to CLEAR, we anticipate that this initiative will assist authors in reporting their radiomics research with greater ease and transparency, as well as editors and reviewers in reviewing manuscripts.Relevance statement Along with the original CLEAR checklist, CLEAR-E3 is expected to provide a more in-depth understanding of the CLEAR items, as well as concrete examples for reporting and evaluating radiomic research.Key points• As a complementary effort to CLEAR, this international collaborative effort aims to assist authors in reporting their radiomics research, as well as editors and reviewers in reviewing radiomics manuscripts.• Based on positive examples from the literature selected by the EuSoMII Radiomics Auditing Group, each CLEAR item explanation was further elaborated in CLEAR-E3.• The resulting explanation and elaboration document with examples can be accessed at https://radiomic.github.io/CLEAR-E3/ .
Collapse
Affiliation(s)
- Burak Kocak
- Department of Radiology, University of Health Sciences, Basaksehir Cam and Sakura City Hospital, Basaksehir, Istanbul, Turkey.
| | - Alessandra Borgheresi
- Department of Clinical, Special and Dental Sciences, University Politecnica delle Marche, Ancona, Italy
- Department of Radiology, University Hospital "Azienda Ospedaliero Universitaria delle Marche", Via Conca 71, 60126, Ancona, Italy
| | - Andrea Ponsiglione
- Department of Advanced Biomedical Sciences, University of Naples Federico II, Naples, Italy
| | - Anna E Andreychenko
- Laboratory for Digital Public Health Technologies, ITMO University, St. Petersburg, Russian Federation
| | - Armando Ugo Cavallo
- Division of Radiology, Istituto Dermopatico dell'Immacolata (IDI) IRCCS, Rome, Italy
| | - Arnaldo Stanzione
- Department of Advanced Biomedical Sciences, University of Naples Federico II, Naples, Italy
| | - Fabio M Doniselli
- Neuroradiology Unit, Fondazione Istituto Neurologico Carlo Besta, Via Celoria 11, 20133, Milano, Italy
| | - Federica Vernuccio
- Section of Radiology, Department of Biomedicine, Neuroscience and Advanced Diagnosis (Bi.N.D), University of Palermo, 90127, Palermo, Italy
| | - Matthaios Triantafyllou
- Department of Medical Imaging, University Hospital of Heraklion, 71110, Crete, Voutes, Greece
| | - Roberto Cannella
- Section of Radiology - Department of Biomedicine, Neuroscience and Advanced Diagnostics (BiND), University of Palermo, Palermo, Italy
| | - Romina Trotta
- Department of Radiology - Fatima Hospital, Seville, Spain
| | | | - Tugba Akinci D'Antonoli
- Institute of Radiology and Nuclear Medicine, Cantonal Hospital Baselland, Liestal, Switzerland
| | - Renato Cuocolo
- Department of Medicine, Surgery and Dentistry, University of Salerno, Baronissi, Italy
| |
Collapse
|
4
|
Kocak B, Akinci D'Antonoli T, Mercaldo N, Alberich-Bayarri A, Baessler B, Ambrosini I, Andreychenko AE, Bakas S, Beets-Tan RGH, Bressem K, Buvat I, Cannella R, Cappellini LA, Cavallo AU, Chepelev LL, Chu LCH, Demircioglu A, deSouza NM, Dietzel M, Fanni SC, Fedorov A, Fournier LS, Giannini V, Girometti R, Groot Lipman KBW, Kalarakis G, Kelly BS, Klontzas ME, Koh DM, Kotter E, Lee HY, Maas M, Marti-Bonmati L, Müller H, Obuchowski N, Orlhac F, Papanikolaou N, Petrash E, Pfaehler E, Pinto Dos Santos D, Ponsiglione A, Sabater S, Sardanelli F, Seeböck P, Sijtsema NM, Stanzione A, Traverso A, Ugga L, Vallières M, van Dijk LV, van Griethuysen JJM, van Hamersvelt RW, van Ooijen P, Vernuccio F, Wang A, Williams S, Witowski J, Zhang Z, Zwanenburg A, Cuocolo R. METhodological RadiomICs Score (METRICS): a quality scoring tool for radiomics research endorsed by EuSoMII. Insights Imaging 2024; 15:8. [PMID: 38228979 PMCID: PMC10792137 DOI: 10.1186/s13244-023-01572-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/01/2023] [Accepted: 11/20/2023] [Indexed: 01/18/2024] Open
Abstract
PURPOSE To propose a new quality scoring tool, METhodological RadiomICs Score (METRICS), to assess and improve research quality of radiomics studies. METHODS We conducted an online modified Delphi study with a group of international experts. It was performed in three consecutive stages: Stage#1, item preparation; Stage#2, panel discussion among EuSoMII Auditing Group members to identify the items to be voted; and Stage#3, four rounds of the modified Delphi exercise by panelists to determine the items eligible for the METRICS and their weights. The consensus threshold was 75%. Based on the median ranks derived from expert panel opinion and their rank-sum based conversion to importance scores, the category and item weights were calculated. RESULT In total, 59 panelists from 19 countries participated in selection and ranking of the items and categories. Final METRICS tool included 30 items within 9 categories. According to their weights, the categories were in descending order of importance: study design, imaging data, image processing and feature extraction, metrics and comparison, testing, feature processing, preparation for modeling, segmentation, and open science. A web application and a repository were developed to streamline the calculation of the METRICS score and to collect feedback from the radiomics community. CONCLUSION In this work, we developed a scoring tool for assessing the methodological quality of the radiomics research, with a large international panel and a modified Delphi protocol. With its conditional format to cover methodological variations, it provides a well-constructed framework for the key methodological concepts to assess the quality of radiomic research papers. CRITICAL RELEVANCE STATEMENT A quality assessment tool, METhodological RadiomICs Score (METRICS), is made available by a large group of international domain experts, with transparent methodology, aiming at evaluating and improving research quality in radiomics and machine learning. KEY POINTS • A methodological scoring tool, METRICS, was developed for assessing the quality of radiomics research, with a large international expert panel and a modified Delphi protocol. • The proposed scoring tool presents expert opinion-based importance weights of categories and items with a transparent methodology for the first time. • METRICS accounts for varying use cases, from handcrafted radiomics to entirely deep learning-based pipelines. • A web application has been developed to help with the calculation of the METRICS score ( https://metricsscore.github.io/metrics/METRICS.html ) and a repository created to collect feedback from the radiomics community ( https://github.com/metricsscore/metrics ).
Collapse
Affiliation(s)
- Burak Kocak
- Department of Radiology, University of Health Sciences, Basaksehir Cam and Sakura City Hospital, Basaksehir, Istanbul, Turkey
| | - Tugba Akinci D'Antonoli
- Institute of Radiology and Nuclear Medicine, Cantonal Hospital Baselland, Liestal, Switzerland.
| | - Nathaniel Mercaldo
- Department of Radiology, Massachusetts General Hospital, Boston, MA, USA
| | | | - Bettina Baessler
- Department of Diagnostic and Interventional Radiology, University Hospital Würzburg, Würzburg, Germany
| | - Ilaria Ambrosini
- Department of Translational Research, Academic Radiology, University of Pisa, Pisa, Italy
| | - Anna E Andreychenko
- Laboratory for Digital Public Health Technologies, ITMO University, St. Petersburg, Russian Federation
| | - Spyridon Bakas
- Division of Computational Pathology, Department of Pathology and Laboratory Medicine, School of Medicine, Indiana University, Indianapolis, IN, USA
- Center for Federated Learning in Precision Medicine, Indiana University, Indianapolis, IN, USA
| | - Regina G H Beets-Tan
- Department of Radiology, The Netherlands Cancer Institute, Amsterdam, the Netherlands
- GROW School for Oncology and Developmental Biology, Maastricht University Medical Center, Maastricht, the Netherlands
- Institute of Regional Health Research, University of Southern Denmark, Odense, Denmark
| | - Keno Bressem
- Department of Radiology, Charité - Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt- Universität zu Berlin, Berlin, Germany
- Berlin Institute of Health at Charité - Universitätsmedizin Berlin, Berlin, Germany
| | - Irene Buvat
- Institut Curie, Inserm, PSL University, Laboratory of Translational Imaging in Oncology, Orsay, France
| | - Roberto Cannella
- Section of Radiology - Department of Biomedicine, Neuroscience and Advanced Diagnostics (BiND), University of Palermo, Palermo, Italy
| | | | - Armando Ugo Cavallo
- Division of Radiology, Istituto Dermopatico dell'Immacolata (IDI) IRCCS, Rome, Italy
| | - Leonid L Chepelev
- Joint Department of Medical Imaging, University Health Network, University of Toronto, Toronto, Canada
| | - Linda Chi Hang Chu
- The Russell H. Morgan Department of Radiology and Radiological Science, Johns Hopkins University School of Medicine, Baltimore, USA
| | - Aydin Demircioglu
- Institute of Diagnostic and Interventional Radiology and Neuroradiology, University Hospital , Essen, Germany
| | - Nandita M deSouza
- Division of Radiotherapy and Imaging, The Institute of Cancer Research, London, UK
- Department of Imaging, The Royal Marsden National Health Service (NHS) Foundation Trust, London, UK
| | - Matthias Dietzel
- Department of Radiology, University Hospital Erlangen, Erlangen, Germany
| | | | - Andrey Fedorov
- Department of Radiology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
| | - Laure S Fournier
- Department of Radiology, Université Paris Cité, AP-HP, Hôpital Européen Georges Pompidou, PARCC UMRS 970, INSERM, Paris, France
| | | | - Rossano Girometti
- Institute of Radiology, Department of Medicine, University of Udine, University Hospital S. Maria della Misericordia, Udine, Italy
| | - Kevin B W Groot Lipman
- Department of Radiology, The Netherlands Cancer Institute, Amsterdam, the Netherlands
- GROW School for Oncology and Developmental Biology, Maastricht University Medical Center, Maastricht, the Netherlands
- Department of Thoracic Oncology, Netherlands Cancer Institute, Amsterdam, the Netherlands
| | - Georgios Kalarakis
- Department of Neuroradiology, Karolinska University Hospital, Stockholm, Sweden
- Department of Clinical Science, Division of Radiology, Intervention and Technology (CLINTEC), Karolinska Institutet, Stockholm, Sweden
- Department of Radiology, Medical School, University of Crete, Heraklion, Greece
| | - Brendan S Kelly
- Department of Radiology, St Vincent's University Hospital, Dublin, Ireland
- Insight Centre for Data Analytics, UCD, Dublin, Ireland
- School of Medicine, University College Dublin, Dublin, Ireland
| | - Michail E Klontzas
- Department of Medical Imaging, University Hospital of Heraklion, Crete, Greece
- Department of Radiology, School of Medicine, University of Crete, Heraklion, Crete, Greece
- Computational Biomedicine Laboratory, Institute of Computer Science, FORTH, Heraklion, Crete, Greece
| | - Dow-Mu Koh
- Department of Radiology, Royal Marsden Hospital, Sutton, UK
| | - Elmar Kotter
- Department of Diagnostic and Interventional Radiology, Faculty of Medicine and Medical Center-University of Freiburg, Freiburg, Germany
| | - Ho Yun Lee
- Department of Radiology and Center for Imaging Science, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul, South Korea
- Department of Health Sciences and Technology, Samsung Advanced Institute for Health Science & Technology (SAIHST), Sungkyunkwan University, Seoul, South Korea
| | - Mario Maas
- Department of Radiology & Nuclear Medicine, Amsterdam UMC Location University of Amsterdam, Meibergdreef 9, Amsterdam, the Netherlands
| | - Luis Marti-Bonmati
- Medical Imaging Department and Biomedical Imaging Research Group, Hospital Universitario y Politécnico La Fe and Health Research Institute, Valencia, Spain
| | - Henning Müller
- University of Applied Sciences of Western Switzerland (HES-SO Valais), Sierra, Switzerland
- Department of Radiology and Medical Informatics, University of Geneva (UniGe), Geneva, Switzerland
| | - Nancy Obuchowski
- Quantitative Health Sciences, Lerner Research Institute, Cleveland Clinic, Cleveland, OH, USA
| | - Fanny Orlhac
- Institut Curie, Inserm, PSL University, Laboratory of Translational Imaging in Oncology, Orsay, France
| | - Nikolaos Papanikolaou
- Computational Clinical Imaging Group, Centre for the Unknown, Champalimaud Foundation, Lisbon, Portugal
- Department of Radiology, Royal Marsden Hospital and The Institute of Cancer Research, London, UK
| | - Ekaterina Petrash
- Radiology department, Research Institute of Pediatric Oncology and Hematology n. a. L.A. Durnov, National Medical Research Center of Oncology n. a. N.N. Blokhin Ministry of Health of Russian Federation, Moscow, Russia
- Medical Department IRA-Labs, Moscow, Russia
| | - Elisabeth Pfaehler
- Institute for advanced simulation (IAS-8): Machine learning and data analytics, Forschungszentrum Jülich, Jülich, Germany
| | - Daniel Pinto Dos Santos
- Department of Radiology, University Hospital of Cologne, Cologne, Germany
- Institute for Diagnostic and Interventional Radiology, Goethe-University Frankfurt Am Main, Frankfurt, Germany
| | - Andrea Ponsiglione
- Department of Advanced Biomedical Sciences, University of Naples Federico II, Naples, Italy
| | - Sebastià Sabater
- Department of Radiation Oncology, Complejo Hospitalario Universitario de Albacete, Albacete, Spain
| | - Francesco Sardanelli
- Department of Biomedical Sciences for Health, Università degli Studi di Milano, Milan, Italy
- Unit of Radiology, IRCCS Policlinico San Donato, San Donato Milanese, Milan, Italy
| | - Philipp Seeböck
- Computational Imaging Research Lab, Department of Biomedical Imaging and Image-guided Therapy, Medical University of Vienna, Vienna, Austria
| | - Nanna M Sijtsema
- Department of Radiation Oncology, University Medical Center Groningen, University of Groningen, Groningen, the Netherlands
| | - Arnaldo Stanzione
- Department of Advanced Biomedical Sciences, University of Naples Federico II, Naples, Italy
| | - Alberto Traverso
- Department of Radiotherapy, Maastro Clinic, Maastricht, the Netherlands
- School of Medicine, Vita-Salute San Raffaele University, Milan, Italy
| | - Lorenzo Ugga
- Department of Advanced Biomedical Sciences, University of Naples Federico II, Naples, Italy
| | - Martin Vallières
- Department of Computer Science, Université de Sherbrooke, Sherbrooke, Canada
- Centre de recherche du Centre hospitalier universitaire de Sherbrooke, Sherbrooke, Canada
| | - Lisanne V van Dijk
- Department of Radiation Oncology, University Medical Center Groningen, University of Groningen, Groningen, the Netherlands
| | | | - Robbert W van Hamersvelt
- Department of Radiology, University Medical Center Utrecht, Utrecht University, Utrecht, the Netherlands
| | - Peter van Ooijen
- Department of Radiotherapy, University of Groningen, University Medical Center Groningen, Groningen, the Netherlands
| | - Federica Vernuccio
- Section of Radiology, Department of Biomedicine, Neuroscience and Advanced Diagnosis (Bi.N.D), University of Palermo, Palermo, 90127, Italy
| | - Alan Wang
- Centre for Medical Imaging & Centre for Brain Research, Faculty of Medical and Health Sciences, Auckland Bioengineering Institute, University of Auckland, Auckland, New Zealand
| | - Stuart Williams
- Department of Radiology, Norfolk & Norwich University Hospital, Colney Lane, Norwich, Norfolk, UK
| | - Jan Witowski
- Department of Radiology, New York University Grossman School of Medicine, New York, USA
| | - Zhongyi Zhang
- School of Information and Communication Technology, Griffith University, Nathan, Brisbane, Australia
| | - Alex Zwanenburg
- National Center for Tumor Diseases (NCT/UCC), Dresden, Germany
- OncoRay - National Center for Radiation Research in Oncology, Faculty of Medicine and University Hospital Carl Gustav Carus, Technische Universität Dresden, Helmholtz-Zentrum Dresden - Rossendorf, Dresden, Germany
- German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Renato Cuocolo
- Department of Medicine, Surgery and Dentistry, University of Salerno, Baronissi, Italy
| |
Collapse
|
5
|
Zhong J, Xing Y, Lu J, Zhang G, Mao S, Chen H, Yin Q, Cen Q, Jiang R, Hu Y, Ding D, Ge X, Zhang H, Yao W. The endorsement of general and artificial intelligence reporting guidelines in radiological journals: a meta-research study. BMC Med Res Methodol 2023; 23:292. [PMID: 38093215 PMCID: PMC10717715 DOI: 10.1186/s12874-023-02117-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/28/2023] [Accepted: 12/01/2023] [Indexed: 12/17/2023] Open
Abstract
BACKGROUND Complete reporting is essential for clinical research. However, the endorsement of reporting guidelines in radiological journals is still unclear. Further, as a field extensively utilizing artificial intelligence (AI), the adoption of both general and AI reporting guidelines would be necessary for enhancing quality and transparency of radiological research. This study aims to investigate the endorsement of general reporting guidelines and those for AI applications in medical imaging in radiological journals, and explore associated journal characteristic variables. METHODS This meta-research study screened journals from the Radiology, Nuclear Medicine & Medical Imaging category, Science Citation Index Expanded of the 2022 Journal Citation Reports, and excluded journals not publishing original research, in non-English languages, and instructions for authors unavailable. The endorsement of fifteen general reporting guidelines and ten AI reporting guidelines was rated using a five-level tool: "active strong", "active weak", "passive moderate", "passive weak", and "none". The association between endorsement and journal characteristic variables was evaluated by logistic regression analysis. RESULTS We included 117 journals. The top-five endorsed reporting guidelines were CONSORT (Consolidated Standards of Reporting Trials, 58.1%, 68/117), PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses, 54.7%, 64/117), STROBE (STrengthening the Reporting of Observational Studies in Epidemiology, 51.3%, 60/117), STARD (Standards for Reporting of Diagnostic Accuracy, 50.4%, 59/117), and ARRIVE (Animal Research Reporting of In Vivo Experiments, 35.9%, 42/117). The most implemented AI reporting guideline was CLAIM (Checklist for Artificial Intelligence in Medical Imaging, 1.7%, 2/117), while other nine AI reporting guidelines were not mentioned. The Journal Impact Factor quartile and publisher were associated with endorsement of reporting guidelines in radiological journals. CONCLUSIONS The general reporting guideline endorsement was suboptimal in radiological journals. The implementation of reporting guidelines for AI applications in medical imaging was extremely low. Their adoption should be strengthened to facilitate quality and transparency of radiological study reporting.
Collapse
Affiliation(s)
- Jingyu Zhong
- Department of Imaging, Tongren Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, 200336, China
| | - Yue Xing
- Department of Imaging, Tongren Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, 200336, China
| | - Junjie Lu
- Department of Epidemiology and Population Health, Stanford University School of Medicine, Stanford, CA, 94305, USA
| | - Guangcheng Zhang
- Department of Orthopedics, Shanghai Sixth People's Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, 200233, China
| | - Shiqi Mao
- Department of Medical Oncology, Shanghai Pulmonary Hospital, Tongji University School of Medicine, Shanghai, 200433, China
| | - Haoda Chen
- Department of General Surgery, Pancreatic Disease Center, Ruijin Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, 200025, China
| | - Qian Yin
- Department of Pathology, Shanghai Sixth People's Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, 200233, China
| | - Qingqing Cen
- Department of Dermatology, Shanghai Ninth People's Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, 200011, China
| | - Run Jiang
- Department of Pharmacovigilance, Shanghai Hansoh BioMedical Co., Ltd., Shanghai, 201203, China
| | - Yangfan Hu
- Department of Imaging, Tongren Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, 200336, China
| | - Defang Ding
- Department of Imaging, Tongren Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, 200336, China
| | - Xiang Ge
- Department of Imaging, Tongren Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, 200336, China
| | - Huan Zhang
- Department of Radiology, Ruijin Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, 200025, China.
| | - Weiwu Yao
- Department of Imaging, Tongren Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, 200336, China.
| |
Collapse
|