1
|
Szabó V, Szabo BT, Orhan K, Veres DS, Manulis D, Ezhov M, Sanders A. Validation of artificial intelligence application for dental caries diagnosis on intraoral bitewing and periapical radiographs. J Dent 2024; 147:105105. [PMID: 38821394 DOI: 10.1016/j.jdent.2024.105105] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/10/2023] [Revised: 05/21/2024] [Accepted: 05/28/2024] [Indexed: 06/02/2024] Open
Abstract
OBJECTIVES This study aimed to assess the reliability of AI-based system that assists the healthcare processes in the diagnosis of caries on intraoral radiographs. METHODS The proximal surfaces of the 323 selected teeth on the intraoral radiographs were evaluated by two independent observers using an AI-based (Diagnocat) system. The presence or absence of carious lesions was recorded during Phase 1. After 4 months, the AI-aided human observers evaluated the same radiographs (Phase 2), and the advanced convolutional neural network (CNN) reassessed the radiographic data (Phase 3). Subsequently, data reflecting human disagreements were excluded (Phase 4). For each phase, the Cohen and Fleiss kappa values, as well as the sensitivity, specificity, positive and negative predictive values, and diagnostic accuracy of Diagnocat, were calculated. RESULTS During the four phases, the range of Cohen kappa values between the human observers and Diagnocat were κ=0.66-1, κ=0.58-0.7, and κ=0.49-0.7. The Fleiss kappa values were κ=0.57-0.8. The sensitivity, specificity and diagnostic accuracy values ranged between 0.51-0.76, 0.88-0.97 and 0.76-0.86, respectively. CONCLUSIONS The Diagnocat CNN supports the evaluation of intraoral radiographs for caries diagnosis, as determined by consensus between human and AI system observers. CLINICAL SIGNIFICANCE Our study may aid in the understanding of deep learning-based systems developed for dental imaging modalities for dentists and contribute to expanding the body of results in the field of AI-supported dental radiology..
Collapse
Affiliation(s)
- Viktor Szabó
- Department of Oral Diagnostics, Faculty of Dentistry, Semmelweis University, Budapest, Hungary
| | - Bence Tamas Szabo
- Department of Oral Diagnostics, Faculty of Dentistry, Semmelweis University, Budapest, Hungary.
| | - Kaan Orhan
- Department of Oral Diagnostics, Faculty of Dentistry, Semmelweis University, Budapest, Hungary; Department of Dentomaxillofacial Radiology, Faculty of Dentistry, Ankara University, Ankara, Turkey; Medical Design Application, and Research Center (MEDITAM), Ankara University, Ankara, Turkey
| | - Dániel Sándor Veres
- Department of Biophysics and Radiation Biology, Semmelweis University, Budapest, Hungary
| | | | | | | |
Collapse
|
2
|
Kocak B, Akinci D'Antonoli T, Ates Kus E, Keles A, Kala A, Kose F, Kadioglu M, Solak S, Sunman S, Temiz ZH. Self-reported checklists and quality scoring tools in radiomics: a meta-research. Eur Radiol 2024:10.1007/s00330-023-10487-5. [PMID: 38180530 DOI: 10.1007/s00330-023-10487-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/24/2023] [Revised: 11/11/2023] [Accepted: 11/24/2023] [Indexed: 01/06/2024]
Abstract
OBJECTIVE To evaluate the use of reporting checklists and quality scoring tools for self-reporting purposes in radiomics literature. METHODS Literature search was conducted in PubMed (date, April 23, 2023). The radiomics literature was sampled at random after a sample size calculation with a priori power analysis. A systematic assessment for self-reporting, including the use of documentation such as completed checklists or quality scoring tools, was conducted in original research papers. These eligible papers underwent independent evaluation by a panel of nine readers, with three readers assigned to each paper. Automatic annotation was used to assist in this process. Then, a detailed item-by-item confirmation analysis was carried out on papers with checklist documentation, with independent evaluation of two readers. RESULTS The sample size calculation yielded 117 papers. Most of the included papers were retrospective (94%; 110/117), single-center (68%; 80/117), based on their private data (89%; 104/117), and lacked external validation (79%; 93/117). Only seven papers (6%) had at least one self-reported document (Radiomics Quality Score (RQS), Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD), or Checklist for Artificial Intelligence in Medical Imaging (CLAIM)), with a statistically significant binomial test (p < 0.001). Median rate of confirmed items for all three documents was 81% (interquartile range, 6). For quality scoring tools, documented scores were higher than suggested scores, with a mean difference of - 7.2 (standard deviation, 6.8). CONCLUSION Radiomic publications often lack self-reported checklists or quality scoring tools. Even when such documents are provided, it is essential to be cautious, as the accuracy of the reported items or scores may be questionable. CLINICAL RELEVANCE STATEMENT Current state of radiomic literature reveals a notable absence of self-reporting with documentation and inaccurate reporting practices. This critical observation may serve as a catalyst for motivating the radiomics community to adopt and utilize such tools appropriately, thereby fostering rigor, transparency, and reproducibility of their research, moving the field forward. KEY POINTS • In radiomics literature, there has been a notable absence of self-reporting with documentation. • Even if such documents are provided, it is critical to exercise caution because the accuracy of the reported items or scores may be questionable. • Radiomics community needs to be motivated to adopt and appropriately utilize the reporting checklists and quality scoring tools.
Collapse
Affiliation(s)
- Burak Kocak
- Department of Radiology, University of Health Sciences, Basaksehir Cam and Sakura City Hospital, Basaksehir, Istanbul, 34480, Turkey.
| | - Tugba Akinci D'Antonoli
- Institute of Radiology and Nuclear Medicine, Cantonal Hospital Baselland, Liestal, Switzerland
| | - Ece Ates Kus
- Department of Neuroradiology, Klinikum Lippe, Lemgo, Germany
| | - Ali Keles
- Department of Radiology, University of Health Sciences, Basaksehir Cam and Sakura City Hospital, Basaksehir, Istanbul, 34480, Turkey
| | - Ahmet Kala
- Department of Radiology, University of Health Sciences, Basaksehir Cam and Sakura City Hospital, Basaksehir, Istanbul, 34480, Turkey
| | - Fadime Kose
- Department of Radiology, University of Health Sciences, Basaksehir Cam and Sakura City Hospital, Basaksehir, Istanbul, 34480, Turkey
| | - Mehmet Kadioglu
- Department of Radiology, University of Health Sciences, Basaksehir Cam and Sakura City Hospital, Basaksehir, Istanbul, 34480, Turkey
| | - Sila Solak
- Department of Radiology, University of Health Sciences, Basaksehir Cam and Sakura City Hospital, Basaksehir, Istanbul, 34480, Turkey
| | - Seyma Sunman
- Department of Radiology, University of Health Sciences, Basaksehir Cam and Sakura City Hospital, Basaksehir, Istanbul, 34480, Turkey
| | - Zisan Hayriye Temiz
- Department of Radiology, University of Health Sciences, Basaksehir Cam and Sakura City Hospital, Basaksehir, Istanbul, 34480, Turkey
| |
Collapse
|
3
|
Zhong J, Xing Y, Lu J, Zhang G, Mao S, Chen H, Yin Q, Cen Q, Jiang R, Hu Y, Ding D, Ge X, Zhang H, Yao W. The endorsement of general and artificial intelligence reporting guidelines in radiological journals: a meta-research study. BMC Med Res Methodol 2023; 23:292. [PMID: 38093215 PMCID: PMC10717715 DOI: 10.1186/s12874-023-02117-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/28/2023] [Accepted: 12/01/2023] [Indexed: 12/17/2023] Open
Abstract
BACKGROUND Complete reporting is essential for clinical research. However, the endorsement of reporting guidelines in radiological journals is still unclear. Further, as a field extensively utilizing artificial intelligence (AI), the adoption of both general and AI reporting guidelines would be necessary for enhancing quality and transparency of radiological research. This study aims to investigate the endorsement of general reporting guidelines and those for AI applications in medical imaging in radiological journals, and explore associated journal characteristic variables. METHODS This meta-research study screened journals from the Radiology, Nuclear Medicine & Medical Imaging category, Science Citation Index Expanded of the 2022 Journal Citation Reports, and excluded journals not publishing original research, in non-English languages, and instructions for authors unavailable. The endorsement of fifteen general reporting guidelines and ten AI reporting guidelines was rated using a five-level tool: "active strong", "active weak", "passive moderate", "passive weak", and "none". The association between endorsement and journal characteristic variables was evaluated by logistic regression analysis. RESULTS We included 117 journals. The top-five endorsed reporting guidelines were CONSORT (Consolidated Standards of Reporting Trials, 58.1%, 68/117), PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses, 54.7%, 64/117), STROBE (STrengthening the Reporting of Observational Studies in Epidemiology, 51.3%, 60/117), STARD (Standards for Reporting of Diagnostic Accuracy, 50.4%, 59/117), and ARRIVE (Animal Research Reporting of In Vivo Experiments, 35.9%, 42/117). The most implemented AI reporting guideline was CLAIM (Checklist for Artificial Intelligence in Medical Imaging, 1.7%, 2/117), while other nine AI reporting guidelines were not mentioned. The Journal Impact Factor quartile and publisher were associated with endorsement of reporting guidelines in radiological journals. CONCLUSIONS The general reporting guideline endorsement was suboptimal in radiological journals. The implementation of reporting guidelines for AI applications in medical imaging was extremely low. Their adoption should be strengthened to facilitate quality and transparency of radiological study reporting.
Collapse
Affiliation(s)
- Jingyu Zhong
- Department of Imaging, Tongren Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, 200336, China
| | - Yue Xing
- Department of Imaging, Tongren Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, 200336, China
| | - Junjie Lu
- Department of Epidemiology and Population Health, Stanford University School of Medicine, Stanford, CA, 94305, USA
| | - Guangcheng Zhang
- Department of Orthopedics, Shanghai Sixth People's Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, 200233, China
| | - Shiqi Mao
- Department of Medical Oncology, Shanghai Pulmonary Hospital, Tongji University School of Medicine, Shanghai, 200433, China
| | - Haoda Chen
- Department of General Surgery, Pancreatic Disease Center, Ruijin Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, 200025, China
| | - Qian Yin
- Department of Pathology, Shanghai Sixth People's Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, 200233, China
| | - Qingqing Cen
- Department of Dermatology, Shanghai Ninth People's Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, 200011, China
| | - Run Jiang
- Department of Pharmacovigilance, Shanghai Hansoh BioMedical Co., Ltd., Shanghai, 201203, China
| | - Yangfan Hu
- Department of Imaging, Tongren Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, 200336, China
| | - Defang Ding
- Department of Imaging, Tongren Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, 200336, China
| | - Xiang Ge
- Department of Imaging, Tongren Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, 200336, China
| | - Huan Zhang
- Department of Radiology, Ruijin Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, 200025, China.
| | - Weiwu Yao
- Department of Imaging, Tongren Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, 200336, China.
| |
Collapse
|
4
|
Ozdag Y, Hayes DS, Callahan C, El Koussaify J, Warnick EP, Foster BK, Klena JC, Grandizio LC. Adherence to Complication Reporting for Randomized Controlled Trials Contained in Clinical Practice Guidelines for the Management of Carpal Tunnel Syndrome. JOURNAL OF HAND SURGERY GLOBAL ONLINE 2023; 5:779-783. [PMID: 38106925 PMCID: PMC10721492 DOI: 10.1016/j.jhsg.2023.06.004] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/08/2023] [Accepted: 06/13/2023] [Indexed: 12/19/2023] Open
Abstract
Purpose Randomized controlled trials (RCTs) are frequently used in creating recommendations contained within clinical practice guidelines (CPGs). However, investigations outside of hand surgery have reported that RCTs within CPGs infrequently report complications and harms-related data. Our purpose was to assess adherence to complication reporting and harms-related outcomes contained in the Consolidated Standards for Reporting (CONSORT) Extension of Harms and Standards for Reporting of Diagnostic Accuracy Studies (STARD) reporting checklists for RCTs within the American Academy of Orthopaedic Surgery (AAOS) CPGs for carpal tunnel syndrome (CTS). Methods We identified all RCTs within the AAOS CTS CPGs. All therapeutic RCTs and diagnostic studies were included. We used the CONSORT Harms Checklist criteria to assess adherence to the reporting of adverse events for therapeutic RCTs and the STARD criteria to assess the diagnostic accuracy of the articles. We defined adequate compliance as adherence to ≥50% of the checklist items. Results We identified 82 therapeutic RCTs and 90 diagnostic accuracy articles within the AAOS CTS CPG. For therapeutic RCTs, we found that the average compliance with the published checklists was 19%. For diagnostic studies, the average compliance with checklists was found to be 55%. Eleven therapeutic RCTs (13%) and 60 diagnostic studies (67%) were determined to have adequate compliance for the CONSORT and STARD checklists, respectively. Conclusions Randomized controlled trials in the AAOS CPGs for CTS have low compliance with the CONSORT Extension for Harms Checklist. Although the overall adherence to the items published in the STARD statement for diagnostic accuracy evaluation remains higher, future efforts should be made to improve the adherence rates to both checklists. Clinical relevance Improved standardization of complication reporting may aid in comparing outcomes across multiple clinical investigations of upper-extremity procedures.
Collapse
Affiliation(s)
- Yagiz Ozdag
- Department of Orthopaedic Surgery, Geisinger Commonwealth School of Medicine, Geisinger Musculoskeletal Institute, Danville, PA
| | - Daniel S. Hayes
- Department of Orthopaedic Surgery, Geisinger Commonwealth School of Medicine, Geisinger Musculoskeletal Institute, Danville, PA
| | - Clarice Callahan
- Department of Orthopaedic Surgery, Geisinger Commonwealth School of Medicine, Geisinger Musculoskeletal Institute, Danville, PA
| | - Jad El Koussaify
- Department of Orthopaedic Surgery, Geisinger Commonwealth School of Medicine, Geisinger Musculoskeletal Institute, Danville, PA
| | - Eugene P. Warnick
- Department of Orthopaedic Surgery, Geisinger Commonwealth School of Medicine, Geisinger Musculoskeletal Institute, Danville, PA
| | - Brian K. Foster
- Department of Orthopaedic Surgery, Geisinger Commonwealth School of Medicine, Geisinger Musculoskeletal Institute, Danville, PA
| | - Joel C. Klena
- Department of Orthopaedic Surgery, Geisinger Commonwealth School of Medicine, Geisinger Musculoskeletal Institute, Danville, PA
| | - Louis C. Grandizio
- Department of Orthopaedic Surgery, Geisinger Commonwealth School of Medicine, Geisinger Musculoskeletal Institute, Danville, PA
| |
Collapse
|
5
|
Rahman E, Philip-Dormston WG, Webb WR, Rao P, Carruthers JD, Carruthers A, Swift A, Goodman GJ, Mosahebi A, Nahai F. Developing Consensus-Based Guidelines for Case Reporting in Aesthetic Medicine: Enhancing Transparency and Standardization. Aesthet Surg J Open Forum 2023; 5:ojad076. [PMID: 37694226 PMCID: PMC10483583 DOI: 10.1093/asjof/ojad076] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 09/12/2023] Open
Abstract
Clinical case reporting plays a vital role in sharing detailed patient narratives, providing insights into rare conditions, innovative treatments, and unexpected outcomes. However, existing reporting guidelines in aesthetic medicine fail to capture the specific nuances of procedures and outcomes in this field. The authors' objectives were to develop comprehensive guidelines for Case REporting in Aesthetic Medicine (CREAM). The study employed a 3-phase consensus process, including a literature review, expert interviews, and a consensus meeting. A diverse group of 10 expert participants (plastic surgeons, dermatologists, noncore specialists, evidence-based medicine expert, and research scientist) in Phase I and 30 experienced aesthetic practitioners in Phase II contributed to the research. Statistical analysis was conducted to assess agreement levels among participants and explore associations and variations within the data. The participants represented various specialties, genders, LGBTQ+ identities, and ethnic backgrounds. The research resulted in the development of the CREAM guidelines, consisting of a 16-item checklist. The guidelines covered essential aspects of case reporting, such as patient and practice information, procedure details, clinical assessment and outcomes, adverse events, and ethical considerations. Statistical analysis indicated a high level of consensus among participants, as well as significant associations between checklist items. CREAM guidelines represent a step toward enhancing transparency and standardization in case reporting in aesthetic medicine. Adhering to these guidelines will allow authors to contribute to a robust evidence base, prioritize patient safety, and drive advancements aesthetic medicine.
Collapse
Affiliation(s)
- Eqram Rahman
- Corresponding Author: Dr Eqram Rahman, Pond Street, Hampstead NW3 2QG, UK. E-mail: ; Instagram: @Beyond_border_Aesthetics
| | | | | | | | | | | | | | | | | | | |
Collapse
|
6
|
Stahl AC, Tietz AS, Dewey M, Kendziora B. Has the quality of reporting improved since it became mandatory to use the Standards for Reporting Diagnostic Accuracy? Insights Imaging 2023; 14:85. [PMID: 37184759 PMCID: PMC10184623 DOI: 10.1186/s13244-023-01432-7] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/10/2023] [Accepted: 04/14/2023] [Indexed: 05/16/2023] Open
Abstract
OBJECTIVES To investigate whether making the Standards for Reporting Diagnostic Accuracy (STARD) mandatory by the leading journal 'Radiology' in 2016 improved the quality of reporting of diagnostic accuracy studies. METHODS A validated search term was used to identify diagnostic accuracy studies published in Radiology in 2015 and 2019. STARD adherence was assessed by two independent reviewers. Each item was scored as yes (1 point) if adequately reported or as no (0 points) if not. The total STARD score per article was calculated. Wilcoxon-Mann-Whitney tests were used to evaluate differences of the total STARD scores between 2015 and 2019. In addition, the total STARD score was compared between studies stratified by study design, citation rate, and data collection. RESULTS The median number of reported STARD items for the total of 66 diagnostic accuracy studies from 2015 to 2019 was 18.5 (interquartile range [IQR] 17.5-20.0) of 29. Adherence to the STARD checklist significantly improved the STARD score from a median of 18.0 (IQR 15.5-19.5) in 2015 to a median of 19.5 (IQR 18.5-21.5) in 2019 (p < 0.001). No significant differences were found between studies stratified by mode of data collection (prospective vs. retrospective studies, p = 0.68), study design (cohort vs. case-control studies, p = 0.81), and citation rate (two groups divided by median split [< 0.56 citations/month vs. ≥ 0.56 citations/month], p = 0.54). CONCLUSIONS Making use of the STARD checklist mandatory significantly increased the adherence with reporting standards for diagnostic accuracy studies and should be considered by editors and publishers for widespread implementation. CRITICAL RELEVANCE STATEMENT Editors may consider making reporting guidelines mandatory to improve the scientific quality.
Collapse
Affiliation(s)
- Ann-Christine Stahl
- Department of Radiology, Charité - Universitätsmedizin Berlin (Corporate Member of Freie Universität Berlin and Humboldt-Universität zu Berlin), Berlin, Germany
| | - Anne-Sophie Tietz
- Department of Radiology, Charité - Universitätsmedizin Berlin (Corporate Member of Freie Universität Berlin and Humboldt-Universität zu Berlin), Berlin, Germany
| | - Marc Dewey
- Department of Radiology, Charité - Universitätsmedizin Berlin (Corporate Member of Freie Universität Berlin and Humboldt-Universität zu Berlin), Berlin, Germany
| | - Benjamin Kendziora
- Department of Radiology, Charité - Universitätsmedizin Berlin (Corporate Member of Freie Universität Berlin and Humboldt-Universität zu Berlin), Berlin, Germany.
- Department of Dermatology and Allergy, University Hospital, Ludwig Maximilian University, Munich, Germany.
| |
Collapse
|
7
|
Kocak B, Baessler B, Bakas S, Cuocolo R, Fedorov A, Maier-Hein L, Mercaldo N, Müller H, Orlhac F, Pinto Dos Santos D, Stanzione A, Ugga L, Zwanenburg A. CheckList for EvaluAtion of Radiomics research (CLEAR): a step-by-step reporting guideline for authors and reviewers endorsed by ESR and EuSoMII. Insights Imaging 2023; 14:75. [PMID: 37142815 PMCID: PMC10160267 DOI: 10.1186/s13244-023-01415-8] [Citation(s) in RCA: 82] [Impact Index Per Article: 82.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/03/2023] [Accepted: 03/24/2023] [Indexed: 05/06/2023] Open
Abstract
Even though radiomics can hold great potential for supporting clinical decision-making, its current use is mostly limited to academic research, without applications in routine clinical practice. The workflow of radiomics is complex due to several methodological steps and nuances, which often leads to inadequate reporting and evaluation, and poor reproducibility. Available reporting guidelines and checklists for artificial intelligence and predictive modeling include relevant good practices, but they are not tailored to radiomic research. There is a clear need for a complete radiomics checklist for study planning, manuscript writing, and evaluation during the review process to facilitate the repeatability and reproducibility of studies. We here present a documentation standard for radiomic research that can guide authors and reviewers. Our motivation is to improve the quality and reliability and, in turn, the reproducibility of radiomic research. We name the checklist CLEAR (CheckList for EvaluAtion of Radiomics research), to convey the idea of being more transparent. With its 58 items, the CLEAR checklist should be considered a standardization tool providing the minimum requirements for presenting clinical radiomics research. In addition to a dynamic online version of the checklist, a public repository has also been set up to allow the radiomics community to comment on the checklist items and adapt the checklist for future versions. Prepared and revised by an international group of experts using a modified Delphi method, we hope the CLEAR checklist will serve well as a single and complete scientific documentation tool for authors and reviewers to improve the radiomics literature.
Collapse
Affiliation(s)
- Burak Kocak
- Department of Radiology, University of Health Sciences, Basaksehir Cam and Sakura City Hospital, Basaksehir, Istanbul, 34480, Turkey.
| | - Bettina Baessler
- Institute of Diagnostic and Interventional Radiology, University Hospital Würzburg, Würzburg, Germany
| | - Spyridon Bakas
- Center for Artificial Intelligence for Integrated Diagnostics (AI2D) & Center for Biomedical Image Computing & Analytics (CBICA), University of Pennsylvania, Philadelphia, PA, USA
- Department of Radiology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
- Department of Pathology and Laboratory Medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
| | - Renato Cuocolo
- Department of Medicine, Surgery, and Dentistry, University of Salerno, Baronissi, Italy
| | - Andrey Fedorov
- Department of Radiology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
| | - Lena Maier-Hein
- Division of Intelligent Medical Systems, German Cancer Research Center, Heidelberg, Germany
- National Center for Tumor Diseases (NCT), Heidelberg, Germany
| | - Nathaniel Mercaldo
- Institute for Technology Assessment, Massachusetts General Hospital, Boston, MA, USA
- Department of Radiology, Massachusetts General Hospital, Boston, MA, USA
| | - Henning Müller
- University of Applied Sciences of Western Switzerland (HES-SO Valais), Valais, Switzerland
- Department of Radiology and Medical Informatics, University of Geneva (UniGe), Geneva, Switzerland
| | - Fanny Orlhac
- Laboratoire d'Imagerie Translationnelle en Oncologie (LITO)-U1288, Institut Curie, Inserm, Université PSL, Orsay, France
| | - Daniel Pinto Dos Santos
- Department of Radiology, University Hospital of Cologne, Cologne, Germany
- Institute for Diagnostic and Interventional Radiology, Goethe-University Frankfurt Am Main, Frankfurt, Germany
| | - Arnaldo Stanzione
- Department of Advanced Biomedical Sciences, University of Naples "Federico II", Naples, Italy
| | - Lorenzo Ugga
- Department of Advanced Biomedical Sciences, University of Naples "Federico II", Naples, Italy
| | - Alex Zwanenburg
- OncoRay-National Center for Radiation Research in Oncology, Faculty of Medicine and University Hospital Carl Gustav Carus, Technische Universität Dresden, Helmholtz-Zentrum Dresden-Rossendorf, Dresden, Germany
- National Center for Tumor Diseases (NCT), Partner Site Dresden, Dresden, Germany
- German Cancer Research Center (DKFZ), Heidelberg, Germany
| |
Collapse
|
8
|
Lennerz JK, Salgado R, Kim GE, Sirintrapun SJ, Thierauf JC, Singh A, Indave I, Bard A, Weissinger SE, Heher YK, de Baca ME, Cree IA, Bennett S, Carobene A, Ozben T, Ritterhouse LL. Diagnostic quality model (DQM): an integrated framework for the assessment of diagnostic quality when using AI/ML. Clin Chem Lab Med 2023; 61:544-557. [PMID: 36696602 DOI: 10.1515/cclm-2022-1151] [Citation(s) in RCA: 7] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/11/2022] [Accepted: 01/13/2023] [Indexed: 01/26/2023]
Abstract
BACKGROUND Laboratory medicine has reached the era where promises of artificial intelligence and machine learning (AI/ML) seem palpable. Currently, the primary responsibility for risk-benefit assessment in clinical practice resides with the medical director. Unfortunately, there is no tool or concept that enables diagnostic quality assessment for the various potential AI/ML applications. Specifically, we noted that an operational definition of laboratory diagnostic quality - for the specific purpose of assessing AI/ML improvements - is currently missing. METHODS A session at the 3rd Strategic Conference of the European Federation of Laboratory Medicine in 2022 on "AI in the Laboratory of the Future" prompted an expert roundtable discussion. Here we present a conceptual diagnostic quality framework for the specific purpose of assessing AI/ML implementations. RESULTS The presented framework is termed diagnostic quality model (DQM) and distinguishes AI/ML improvements at the test, procedure, laboratory, or healthcare ecosystem level. The operational definition illustrates the nested relationship among these levels. The model can help to define relevant objectives for implementation and how levels come together to form coherent diagnostics. The affected levels are referred to as scope and we provide a rubric to quantify AI/ML improvements while complying with existing, mandated regulatory standards. We present 4 relevant clinical scenarios including multi-modal diagnostics and compare the model to existing quality management systems. CONCLUSIONS A diagnostic quality model is essential to navigate the complexities of clinical AI/ML implementations. The presented diagnostic quality framework can help to specify and communicate the key implications of AI/ML solutions in laboratory diagnostics.
Collapse
Affiliation(s)
- Jochen K Lennerz
- Department of Pathology, Massachusetts General Hospital/Harvard Medical, Boston, MA, USA
| | - Roberto Salgado
- Department of Pathology, GZA-ZNA Hospitals, Antwerp, Belgium
- Division of Research, Peter Mac Callum Cancer Centre, Melbourne, Australia
| | - Grace E Kim
- Department of Pathology, University of California San Francisco, San Francisco, CA, USA
| | | | - Julia C Thierauf
- Department of Pathology, Massachusetts General Hospital/Harvard Medical, Boston, MA, USA
- Department of Otorhinolaryngology, Head and Neck Surgery, German Cancer Research Center (DKFZ), Heidelberg University Hospital and Research Group Molecular Mechanisms of Head and Neck Tumors, Heidelberg, Germany
| | - Ankit Singh
- Department of Pathology, Massachusetts General Hospital/Harvard Medical, Boston, MA, USA
| | - Iciar Indave
- European Monitoring Centre for Drugs and Drug Addiction (EMCDDA), Lisbon, Portugal
| | - Adam Bard
- Department of Pathology, Massachusetts General Hospital/Harvard Medical, Boston, MA, USA
| | | | - Yael K Heher
- Department of Pathology, Massachusetts General Hospital/Harvard Medical, Boston, MA, USA
| | | | - Ian A Cree
- International Agency for Research on Cancer (IARC), World Health Organization, Lyon, France
| | - Shannon Bennett
- Department of Laboratory Medicine and Pathology (DLMP), Mayo Clinic, Rochester, MN, USA
| | - Anna Carobene
- IRCCS San Raffaele Scientific Institute, Milan, Italy
| | - Tomris Ozben
- Medical Faculty, Dept. of Clinical Biochemistry, Akdeniz University, Antalya, Türkiye
- Medical Faculty, Clinical and Experimental Medicine, Ph.D. Program, University of Modena and Reggio Emilia, Modena, Italy
| | - Lauren L Ritterhouse
- Department of Pathology, Massachusetts General Hospital/Harvard Medical, Boston, MA, USA
| |
Collapse
|
9
|
Quality assessment of machine learning models for diagnostic imaging in orthopaedics: A systematic review. Artif Intell Med 2022; 132:102396. [DOI: 10.1016/j.artmed.2022.102396] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/29/2022] [Revised: 08/30/2022] [Accepted: 08/30/2022] [Indexed: 01/17/2023]
|