|
1
|
REINKE ANNIKA, TIZABI MINUD, BAUMGARTNER MICHAEL, EISENMANN MATTHIAS, HECKMANN-NÖTZEL DOREEN, KAVUR AEMRE, RÄDSCH TIM, SUDRE CAROLEH, ACION LAURA, ANTONELLI MICHELA, ARBEL TAL, BAKAS SPYRIDON, BENIS ARRIEL, BLASCHKO MATTHEWB, BUETTNER FLORIAN, CARDOSO MJORGE, CHEPLYGINA VERONIKA, CHEN JIANXU, CHRISTODOULOU EVANGELIA, CIMINI BETHA, COLLINS GARYS, FARAHANI KEYVAN, FERRER LUCIANA, GALDRAN ADRIAN, VAN GINNEKEN BRAM, GLOCKER BEN, GODAU PATRICK, HAASE ROBERT, HASHIMOTO DANIELA, HOFFMAN MICHAELM, HUISMAN MEREL, ISENSEE FABIAN, JANNIN PIERRE, KAHN CHARLESE, KAINMUELLER DAGMAR, KAINZ BERNHARD, KARARGYRIS ALEXANDROS, KARTHIKESALINGAM ALAN, KENNGOTT HANNES, KLEESIEK JENS, KOFLER FLORIAN, KOOI THIJS, KOPP-SCHNEIDER ANNETTE, KOZUBEK MICHAL, KRESHUK ANNA, KURC TAHSIN, LANDMAN BENNETTA, LITJENS GEERT, MADANI AMIN, MAIER-HEIN KLAUS, MARTEL ANNEL, MATTSON PETER, MEIJERING ERIK, MENZE BJOERN, MOONS KARELG, MÜLLER HENNING, NICHYPORUK BRENNAN, NICKEL FELIX, PETERSEN JENS, RAFELSKI SUSANNEM, RAJPOOT NASIR, REYES MAURICIO, RIEGLER MICHAELA, RIEKE NICOLA, SAEZ-RODRIGUEZ JULIO, SÁNCHEZ CLARAI, SHETTY SHRAVYA, SUMMERS RONALDM, TAHA ABDELA, TIULPIN ALEKSEI, TSAFTARIS SOTIRIOSA, VAN CALSTER BEN, VAROQUAUX GAËL, YANIV ZIVR, JÄGER PAULF, MAIER-HEIN LENA. Understanding metric-related pitfalls in image analysis validation. ArXiv 2024:arXiv:2302.01790v4. [PMID: 36945687 PMCID: PMC10029046] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Download PDF] [Subscribe] [Scholar Register] [Indexed: 03/23/2023]
Abstract
Validation metrics are key for the reliable tracking of scientific progress and for bridging the current chasm between artificial intelligence (AI) research and its translation into practice. However, increasing evidence shows that particularly in image analysis, metrics are often chosen inadequately in relation to the underlying research problem. This could be attributed to a lack of accessibility of metric-related knowledge: While taking into account the individual strengths, weaknesses, and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multi-stage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides the first reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Focusing on biomedical image analysis but with the potential of transfer to other fields, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. To facilitate comprehension, illustrations and specific examples accompany each pitfall. As a structured body of information accessible to researchers of all levels of expertise, this work enhances global comprehension of a key topic in image analysis validation.
Collapse
Affiliation(s)
- ANNIKA REINKE
- German Cancer Research Center (DKFZ) Heidelberg, Division of Intelligent Medical Systems and HI Helmholtz Imaging, Germany and Faculty of Mathematics and Computer Science, Heidelberg University, Heidelberg, Germany
| | - MINU D. TIZABI
- German Cancer Research Center (DKFZ) Heidelberg, Division of Intelligent Medical Systems, Germany and National Center for Tumor Diseases (NCT), NCT Heidelberg, a partnership between DKFZ and University Medical Center Heidelberg, Germany
| | - MICHAEL BAUMGARTNER
- German Cancer Research Center (DKFZ) Heidelberg, Division of Medical Image Computing, Germany
| | - MATTHIAS EISENMANN
- German Cancer Research Center (DKFZ) Heidelberg, Division of Intelligent Medical Systems, Germany
| | - DOREEN HECKMANN-NÖTZEL
- German Cancer Research Center (DKFZ) Heidelberg, Division of Intelligent Medical Systems, Germany and National Center for Tumor Diseases (NCT), NCT Heidelberg, a partnership between DKFZ and University Medical Center Heidelberg, Germany
| | - A. EMRE KAVUR
- HI Applied Computer Vision Lab, Division of Medical Image Computing; German Cancer Research Center (DKFZ) Heidelberg, Division of Intelligent Medical Systems, Germany
| | - TIM RÄDSCH
- German Cancer Research Center (DKFZ) Heidelberg, Division of Intelligent Medical Systems and HI Helmholtz Imaging, Germany
| | - CAROLE H. SUDRE
- MRC Unit for Lifelong Health and Ageing at UCL and Centre for Medical Image Computing, Department of Computer Science, University College London, London, UK and School of Biomedical Engineering and Imaging Science, King’s College London, London, UK
| | - LAURA ACION
- Instituto de Cálculo, CONICET – Universidad de Buenos Aires, Buenos Aires, Argentina
| | - MICHELA ANTONELLI
- School of Biomedical Engineering and Imaging Science, King’s College London, London, UK and Centre for Medical Image Computing, University College London, London, UK
| | - TAL ARBEL
- Centre for Intelligent Machines and MILA (Quebec Artificial Intelligence Institute), McGill University, Montreal, Canada
| | - SPYRIDON BAKAS
- Division of Computational Pathology, Dept of Pathology & Laboratory Medicine, Indiana University School of Medicine, IU Health Information and Translational Sciences Building, Indianapolis, USA and Center for Biomedical Image Computing and Analytics (CBICA), University of Pennsylvania, Richards Medical Research Laboratories FL7, Philadelphia, PA, USA
| | - ARRIEL BENIS
- Department of Digital Medical Technologies, Holon Institute of Technology, Holon, Israel and European Federation for Medical Informatics, Le Mont-sur-Lausanne, Switzerland
| | - MATTHEW B. BLASCHKO
- Center for Processing Speech and Images, Department of Electrical Engineering, KU Leuven, Leuven, Belgium
| | - FLORIAN BUETTNER
- German Cancer Consortium (DKTK), partner site Frankfurt/Mainz, a partnership between DKFZ and UCT Frankfurt-Marburg, Germany, German Cancer Research Center (DKFZ) Heidelberg, Germany, Goethe University Frankfurt, Department of Medicine, Germany, Goethe University Frankfurt, Department of Informatics, Germany, and Frankfurt Cancer Insititute, Germany
| | - M. JORGE CARDOSO
- School of Biomedical Engineering and Imaging Science, King’s College London, London, UK
| | - VERONIKA CHEPLYGINA
- Department of Computer Science, IT University of Copenhagen, Copenhagen, Denmark
| | - JIANXU CHEN
- Leibniz-Institut für Analytische Wissenschaften – ISAS – e.V., Dortmund, Germany
| | - EVANGELIA CHRISTODOULOU
- German Cancer Research Center (DKFZ) Heidelberg, Division of Intelligent Medical Systems, Germany
| | - BETH A. CIMINI
- Imaging Platform, Broad Institute of MIT and Harvard, Cambridge, Massachusetts, USA
| | - GARY S. COLLINS
- Centre for Statistics in Medicine, University of Oxford, Oxford, UK
| | - KEYVAN FARAHANI
- Center for Biomedical Informatics and Information Technology, National Cancer Institute, Bethesda, MD, USA
| | - LUCIANA FERRER
- Instituto de Investigación en Ciencias de la Computación (ICC), CONICET-UBA, Ciudad Universitaria, Ciudad Autónoma de Buenos Aires, Argentina
| | - ADRIAN GALDRAN
- Universitat Pompeu Fabra, Barcelona, Spain and University of Adelaide, Adelaide, Australia
| | - BRAM VAN GINNEKEN
- Fraunhofer MEVIS, Bremen, Germany and Radboud Institute for Health Sciences, Radboud University Medical Center, Nijmegen, The Netherlands
| | - BEN GLOCKER
- Department of Computing, Imperial College London, London, UK
| | - PATRICK GODAU
- German Cancer Research Center (DKFZ) Heidelberg, Division of Intelligent Medical Systems, Germany, Faculty of Mathematics and Computer Science, Heidelberg University, Heidelberg, Germany, and National Center for Tumor Diseases (NCT), NCT Heidelberg, a partnership between DKFZ and University Medical Center Heidelberg, Germany
| | - ROBERT HAASE
- Now with: Center for Scalable Data Analytics and Artificial Intelligence (ScaDS.AI), Leipzig University, Leipzig, Germany, DFG Cluster of Excellence “Physics of Life”, Technische Universität (TU) Dresden, Dresden, Germany, and Center for Systems Biology , Dresden, Germany
| | - DANIEL A. HASHIMOTO
- Department of Surgery, Perelman School of Medicine, Philadelphia, PA, USA and General Robotics Automation Sensing and Perception Laboratory, School of Engineering and Applied Science, University of Pennsylvania, Philadelphia, PA, USA
| | - MICHAEL M. HOFFMAN
- Princess Margaret Cancer Centre, University Health Network, Toronto, Canada, Department of Medical Biophysics, University of Toronto, Toronto, Canada, Department of Computer Science, University of Toronto, Toronto, Canada, and Vector Institute for Artificial Intelligence, Toronto, Canada
| | - MEREL HUISMAN
- Department of Radiology and Nuclear Medicine, Radboud University Medical Center, Nijmegen, The Netherlands
| | - FABIAN ISENSEE
- German Cancer Research Center (DKFZ) Heidelberg, Division of Medical Image Computing and HI Applied Computer Vision Lab, Germany
| | - PIERRE JANNIN
- Laboratoire Traitement du Signal et de l’Image – UMR_S 1099, Université de Rennes 1, Rennes, France and INSERM, Paris Cedex, France
| | - CHARLES E. KAHN
- Department of Radiology and Institute for Biomedical Informatics, University of Pennsylvania, Philadelphia, PA, USA
| | - DAGMAR KAINMUELLER
- Max-Delbrück Center for Molecular Medicine in the Helmholtz Association (MDC), Biomedical Image Analysis and HI Helmholtz Imaging, Berlin, Germany and University of Potsdam, Digital Engineering Faculty, Potsdam, Germany
| | - BERNHARD KAINZ
- Department of Computing, Faculty of Engineering, Imperial College London, London, UK and Department AIBE, Friedrich-Alexander-Universität (FAU), Erlangen-Nürnberg, Germany
| | | | | | - HANNES KENNGOTT
- Department of General, Visceral and Transplantation Surgery, Heidelberg University Hospital, Heidelberg, Germany
| | - JENS KLEESIEK
- Translational Image-guided Oncology (TIO), Institute for AI in Medicine (IKIM), University Medicine Essen, Essen, Germany
| | | | | | | | - MICHAL KOZUBEK
- Centre for Biomedical Image Analysis and Faculty of Informatics, Masaryk University, Brno, Czech Republic
| | - ANNA KRESHUK
- Cell Biology and Biophysics Unit, European Molecular Biology Laboratory (EMBL), Heidelberg, Germany
| | - TAHSIN KURC
- Department of Biomedical Informatics, Stony Brook University, Stony Brook, NY, USA
| | | | - GEERT LITJENS
- Department of Pathology, Radboud University Medical Center, Nijmegen, The Netherlands
| | - AMIN MADANI
- Department of Surgery, University Health Network, Philadelphia, PA, Canada
| | - KLAUS MAIER-HEIN
- German Cancer Research Center (DKFZ) Heidelberg, Division of Medical Image Computing and HI Helmholtz Imaging, Germany and Pattern Analysis and Learning Group, Department of Radiation Oncology, Heidelberg University Hospital, Heidelberg, Germany
| | - ANNE L. MARTEL
- Physical Sciences, Sunnybrook Research Institute, Toronto, Canada and Department of Medical Biophysics, University of Toronto, Toronto, ON, Canada
| | | | - ERIK MEIJERING
- School of Computer Science and Engineering, University of New South Wales, Sydney, Australia
| | - BJOERN MENZE
- Department of Quantitative Biomedicine, University of Zurich, Zurich, Switzerland
| | - KAREL G.M. MOONS
- Julius Center for Health Sciences and Primary Care, UMC Utrecht, Utrecht University, Utrecht, The Netherlands
| | - HENNING MÜLLER
- Information Systems Institute, University of Applied Sciences Western Switzerland (HES-SO), Sierre, Switzerland and Medical Faculty, University of Geneva, Geneva, Switzerland
| | | | - FELIX NICKEL
- Department of General, Visceral and Thoracic Surgery, University Medical Center Hamburg-Eppendorf, Hamburg, Germany
| | - JENS PETERSEN
- German Cancer Research Center (DKFZ) Heidelberg, Division of Medical Image Computing, Germany
| | | | - NASIR RAJPOOT
- Tissue Image Analytics Laboratory, Department of Computer Science, University of Warwick, Coventry, UK
| | - MAURICIO REYES
- ARTORG Center for Biomedical Engineering Research, University of Bern, Bern, Switzerland and Department of Radiation Oncology, University Hospital Bern, University of Bern, Bern, Switzerland
| | - MICHAEL A. RIEGLER
- Simula Metropolitan Center for Digital Engineering, Oslo, Norway and UiT The Arctic University of Norway, Tromsø, Norway
| | | | - JULIO SAEZ-RODRIGUEZ
- Institute for Computational Biomedicine, Heidelberg University, Heidelberg. Germany and Faculty of Medicine, Heidelberg University Hospital, Heidelberg, Germany
| | - CLARA I. SÁNCHEZ
- Informatics Institute, Faculty of Science, University of Amsterdam, Amsterdam, The Netherlands
| | | | | | - ABDEL A. TAHA
- Institute of Information Systems Engineering, TU Wien, Vienna, Austria
| | - ALEKSEI TIULPIN
- Research Unit of Health Sciences and Technology, Faculty of Medicine, University of Oulu, Oulu, Finland and Neurocenter Oulu, Oulu University Hospital, Oulu, Finland
| | | | - BEN VAN CALSTER
- Department of Development and Regeneration and EPI-centre, KU Leuven, Leuven, Belgium and Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, The Netherlands
| | - GAËL VAROQUAUX
- Parietal project team, INRIA Saclay-Île de France, Palaiseau, France
| | - ZIV R. YANIV
- National Institute of Allergy and Infectious Diseases, National Institutes of Health, Bethesda, MD, USA
| | - PAUL F. JÄGER
- German Cancer Research Center (DKFZ) Heidelberg, Interactive Machine Learning Group and HI Helmholtz Imaging, Germany
| | - LENA MAIER-HEIN
- German Cancer Research Center (DKFZ) Heidelberg, Division of Intelligent Medical Systems and HI Helmholtz Imaging, Germany, Faculty of Mathematics and Computer Science and Medical Faculty, Heidelberg University, Heidelberg, Germany, and National Center for Tumor Diseases (NCT), NCT Heidelberg, a partnership between DKFZ and University Medical Center Heidelberg, Germany
| |
Collapse
|