1
|
Elfer K, Gardecki E, Garcia V, Ly A, Hytopoulos E, Wen S, Hanna MG, Peeters DJE, Saltz J, Ehinger A, Dudgeon SN, Li X, Blenman KRM, Chen W, Green U, Birmingham R, Pan T, Lennerz JK, Salgado R, Gallas BD. Reproducible Reporting of the Collection and Evaluation of Annotations for Artificial Intelligence Models. Mod Pathol 2024; 37:100439. [PMID: 38286221 DOI: 10.1016/j.modpat.2024.100439] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/18/2023] [Revised: 12/14/2023] [Accepted: 01/21/2024] [Indexed: 01/31/2024]
Abstract
This work puts forth and demonstrates the utility of a reporting framework for collecting and evaluating annotations of medical images used for training and testing artificial intelligence (AI) models in assisting detection and diagnosis. AI has unique reporting requirements, as shown by the AI extensions to the Consolidated Standards of Reporting Trials (CONSORT) and Standard Protocol Items: Recommendations for Interventional Trials (SPIRIT) checklists and the proposed AI extensions to the Standards for Reporting Diagnostic Accuracy (STARD) and Transparent Reporting of a Multivariable Prediction model for Individual Prognosis or Diagnosis (TRIPOD) checklists. AI for detection and/or diagnostic image analysis requires complete, reproducible, and transparent reporting of the annotations and metadata used in training and testing data sets. In an earlier work by other researchers, an annotation workflow and quality checklist for computational pathology annotations were proposed. In this manuscript, we operationalize this workflow into an evaluable quality checklist that applies to any reader-interpreted medical images, and we demonstrate its use for an annotation effort in digital pathology. We refer to this quality framework as the Collection and Evaluation of Annotations for Reproducible Reporting of Artificial Intelligence (CLEARR-AI).
Collapse
Affiliation(s)
- Katherine Elfer
- United States Food and Drug Administration, Center for Devices and Radiological Health, Office of Science and Engineering Laboratories, Division of Imaging Diagnostics and Software Reliability, Silver Spring, Maryland; National Institutes of Health, National Cancer Institute, Division of Cancer Prevention, Cancer Prevention Fellowship Program, Bethesda, Maryland.
| | - Emma Gardecki
- United States Food and Drug Administration, Center for Devices and Radiological Health, Office of Science and Engineering Laboratories, Division of Imaging Diagnostics and Software Reliability, Silver Spring, Maryland
| | - Victor Garcia
- United States Food and Drug Administration, Center for Devices and Radiological Health, Office of Science and Engineering Laboratories, Division of Imaging Diagnostics and Software Reliability, Silver Spring, Maryland
| | - Amy Ly
- Department of Pathology, Massachusetts General Hospital, Boston, Massachusetts
| | | | - Si Wen
- United States Food and Drug Administration, Center for Devices and Radiological Health, Office of Science and Engineering Laboratories, Division of Imaging Diagnostics and Software Reliability, Silver Spring, Maryland
| | - Matthew G Hanna
- Department of Pathology and Laboratory Medicine, Memorial Sloan Kettering Cancer Center, New York, New York
| | - Dieter J E Peeters
- Department of Pathology, University Hospital Antwerp/University of Antwerp, Antwerp, Belgium; Department of Pathology, Sint-Maarten Hospital, Mechelen, Belgium
| | - Joel Saltz
- Department of Biomedical Informatics, Stony Brook University, Stony Brook, New York
| | - Anna Ehinger
- Department of Clinical Genetics, Pathology and Molecular Diagnostics, Laboratory Medicine, Lund University, Lund, Sweden
| | - Sarah N Dudgeon
- Department of Laboratory Medicine, Yale School of Medicine, New Haven, Connecticut
| | - Xiaoxian Li
- Department of Pathology and Laboratory Medicine, Emory University School of Medicine, Atlanta, Georgia
| | - Kim R M Blenman
- Department of Internal Medicine, Section of Medical Oncology, Yale School of Medicine and Yale Cancer Center, Yale University, New Haven, Connecticut; Department of Computer Science, School of Engineering and Applied Science, Yale University, New Haven, Connecticut
| | - Weijie Chen
- United States Food and Drug Administration, Center for Devices and Radiological Health, Office of Science and Engineering Laboratories, Division of Imaging Diagnostics and Software Reliability, Silver Spring, Maryland
| | - Ursula Green
- Department of Biomedical Informatics, Emory University School of Medicine, Atlanta, Georgia
| | - Ryan Birmingham
- United States Food and Drug Administration, Center for Devices and Radiological Health, Office of Science and Engineering Laboratories, Division of Imaging Diagnostics and Software Reliability, Silver Spring, Maryland; Department of Biomedical Informatics, Emory University School of Medicine, Atlanta, Georgia
| | - Tony Pan
- Department of Biomedical Informatics, Emory University School of Medicine, Atlanta, Georgia
| | - Jochen K Lennerz
- Department of Pathology, Center for Integrated Diagnostics, Massachusetts General Hospital, Harvard Medical School, Boston, Massachusetts
| | - Roberto Salgado
- Division of Research, Peter Mac Callum Cancer Centre, Melbourne, Australia; Department of Pathology, GZA-ZNA Hospitals, Antwerp, Belgium
| | - Brandon D Gallas
- United States Food and Drug Administration, Center for Devices and Radiological Health, Office of Science and Engineering Laboratories, Division of Imaging Diagnostics and Software Reliability, Silver Spring, Maryland
| |
Collapse
|
2
|
Hart S, Garcia V, Dudgeon SN, Hanna MG, Li X, Blenman KRM, Elfer K, Ly A, Salgado R, Saltz J, Gupta R, Hytopoulos E, Larsimont D, Lennerz J, Gallas BD. Initial interactions with the FDA on developing a validation dataset as a medical device development tool. J Pathol 2023; 261:378-384. [PMID: 37794720 PMCID: PMC10841854 DOI: 10.1002/path.6208] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/09/2023] [Revised: 07/14/2023] [Accepted: 08/24/2023] [Indexed: 10/06/2023]
Abstract
Quantifying tumor-infiltrating lymphocytes (TILs) in breast cancer tumors is a challenging task for pathologists. With the advent of whole slide imaging that digitizes glass slides, it is possible to apply computational models to quantify TILs for pathologists. Development of computational models requires significant time, expertise, consensus, and investment. To reduce this burden, we are preparing a dataset for developers to validate their models and a proposal to the Medical Device Development Tool (MDDT) program in the Center for Devices and Radiological Health of the U.S. Food and Drug Administration (FDA). If the FDA qualifies the dataset for its submitted context of use, model developers can use it in a regulatory submission within the qualified context of use without additional documentation. Our dataset aims at reducing the regulatory burden placed on developers of models that estimate the density of TILs and will allow head-to-head comparison of multiple computational models on the same data. In this paper, we discuss the MDDT preparation and submission process, including the feedback we received from our initial interactions with the FDA and propose how a qualified MDDT validation dataset could be a mechanism for open, fair, and consistent measures of computational model performance. Our experiences will help the community understand what the FDA considers relevant and appropriate (from the perspective of the submitter), at the early stages of the MDDT submission process, for validating stromal TIL density estimation models and other potential computational models. © 2023 The Authors. The Journal of Pathology published by John Wiley & Sons Ltd on behalf of The Pathological Society of Great Britain and Ireland. This article has been contributed to by U.S. Government employees and their work is in the public domain in the USA.
Collapse
Affiliation(s)
- Steven Hart
- Department of Laboratory Medicine and Pathology, Mayo Clinic, Rochester MN, USA
| | - Victor Garcia
- Division of Imaging, Diagnostics, and Software Reliability, Office Science and Engineering Laboratories, Center for Devices and Radiological Health, US Food and Drug Administration, Silver Spring, MD, USA
| | - Sarah N. Dudgeon
- Computational Biology and Bioinformatics Program, Yale University, New Haven, CT, USA
| | | | - Xiaoxian Li
- Department of Pathology and Laboratory Medicine, Emory University, Atlanta, GA, USA
| | - Kim RM Blenman
- Department of Internal Medicine, Section of Medical Oncology, School of Medicine, Yale University, New Haven, CT, USA
- Department of Computer Science, School of Engineering and Applied Science, Yale University, New Haven, CT, USA
| | - Katherine Elfer
- Division of Imaging, Diagnostics, and Software Reliability, Office Science and Engineering Laboratories, Center for Devices and Radiological Health, US Food and Drug Administration, Silver Spring, MD, USA
| | - Amy Ly
- Department of Pathology, Massachusetts General Hospital, MA, USA
| | - Roberto Salgado
- Department of Pathology, GZA-ZNA Hospitals, Antwerp, Belgium
- Division of Research, Peter Mac Callum Cancer Centre, Melbourne, Australia
| | - Joel Saltz
- Department of Biomedical Informatics, Stony Brook School of Medicine, Stony Brook NY, USA
| | - Rajarsi Gupta
- Department of Biomedical Informatics, Stony Brook School of Medicine, Stony Brook NY, USA
| | | | - Denis Larsimont
- Department of Pathology, Institut Jules Bordet, Université Libre de Bruxelles, Brussels, Belgium
| | - Jochen Lennerz
- Massachusetts General Hospital/Massachusetts General Hospital, Center for Integrated Diagnostics, Boston, MA, USA
| | - Brandon D. Gallas
- Division of Imaging, Diagnostics, and Software Reliability, Office Science and Engineering Laboratories, Center for Devices and Radiological Health, US Food and Drug Administration, Silver Spring, MD, USA
| |
Collapse
|