Understanding metric-related pitfalls in image analysis validation
Annika Reinke
(1)
,
Minu D. Tizabi
(1)
,
Michael Baumgartner
(1)
,
Matthias Eisenmann
(1)
,
Doreen Heckmann-Nötzel
(1)
,
A. Emre Kavur
(1)
,
Tim Rädsch
(1)
,
Carole H. Sudre
(2)
,
Laura Acion
(3)
,
Michela Antonelli
(4)
,
Tal Arbel
(5, 6)
,
Spyridon Bakas
(7)
,
Arriel Benis
(8)
,
Matthew Blaschko
(9)
,
Florian Buettner
(10)
,
M. Jorge Cardoso
(4)
,
Veronika Cheplygina
(11)
,
Jianxu Chen
(12)
,
Evangelia Christodoulou
(1)
,
Beth A. Cimini
(13)
,
Gary S. Collins
(14)
,
Keyvan Farahani
(15)
,
Luciana Ferrer
(3)
,
Adrian Galdran
(16)
,
Bram van Ginneken
(17)
,
Ben Glocker
(18)
,
Patrick Godau
(1)
,
Robert Haase
(19)
,
Daniel A. Hashimoto
(20)
,
Michael M. Hoffman
(21)
,
Merel Huisman
(22)
,
Fabian Isensee
(1)
,
Pierre Jannin
(23, 24)
,
Charles E. Kahn
(25)
,
Dagmar Kainmueller
(26)
,
Bernhard Kainz
(18)
,
Alexandros Karargyris
(27)
,
Alan Karthikesalingam
(28)
,
Hannes Kenngott
(29)
,
Jens Kleesiek
(30)
,
Florian Kofler
(31)
,
Thijs Kooi
(32)
,
Annette Kopp-Schneider
(1)
,
Michal Kozubek
(33)
,
Anna Kreshuk
(34)
,
Tahsin Kurc
(35)
,
Bennett A. Landman
(36)
,
Geert Litjens
(37)
,
Amin Madani
(38)
,
Klaus Maier-Hein
(1)
,
Anne L. Martel
(39)
,
Peter Mattson
(28)
,
Erik Meijering
(40)
,
Bjoern Menze
(41)
,
Karel G. M. Moons
(42)
,
Henning Müller
(43)
,
Brennan Nichyporuk
(5)
,
Felix Nickel
(44)
,
Jens Petersen
(1)
,
Susanne M. Rafelski
(45)
,
Nasir Rajpoot
(46)
,
Mauricio Reyes
(47)
,
Michael A. Riegler
(48)
,
Nicola Rieke
(49)
,
Julio Saez-Rodriguez
(50)
,
Clara I. Sánchez
(51)
,
Shravya Shetty
(28)
,
Ronald M. Summers
(52)
,
Abdel A. Taha
(53)
,
Aleksei Tiulpin
(54)
,
Sotirios A. Tsaftaris
(55)
,
Ben van Calster
(56)
,
Gaël Varoquaux
(57, 58)
,
Ziv R. Yaniv
(59)
,
Paul F. Jäger
(1)
,
Lena Maier-Hein
(1)
1
DKFZ -
German Cancer Research Center - Deutsches Krebsforschungszentrum [Heidelberg]
2 UCL - University College of London [London]
3 CONICET - Consejo Nacional de Investigaciones Científicas y Técnicas [Buenos Aires]
4 King‘s College London
5 Mila - Institut québécois d’intelligence artificielle
6 McGill University = Université McGill [Montréal, Canada]
7 Indiana University School of Medicine
8 HIT - Holon Institut of Technology
9 KU-ESAT - Department of Electrical Engineering [KU Leuven]
10 DKTK - German Cancer Consortium [Heidelberg]
11 ITU - IT University of Copenhagen
12 ISAS - Leibniz Institute for Analytical Sciences
13 Broad Institute [Cambridge]
14 University of Oxford
15 NCI-NIH - National Cancer Institute [Bethesda]
16 UPF - Universitat Pompeu Fabra [Barcelona]
17 Fraunhofer MEVIS - Fraunhofer Institute for Digital Medicine
18 Imperial College London
19 Leipzig University / Universität Leipzig
20 Perelman School of Medicine
21 University of Toronto
22 Radboud University Medical Center [Nijmegen]
23 LTSI - Laboratoire Traitement du Signal et de l'Image
24 Centre Hospitalier Universitaire de Rennes [CHU Rennes] = Rennes University Hospital [Ponchaillou]
25 University of Pennsylvania
26 University of Potsdam = Universität Potsdam
27 IHU Strasbourg - L'Institut hospitalo-universitaire de Strasbourg
28 Google Inc
29 Heidelberg University Hospital [Heidelberg]
30 AöR - University Hospital Essen
31 Helmholtz AI
32 SNU - Seoul National University [Seoul]
33 MUNI - Masaryk University [Brno]
34 EMBL Heidelberg
35 SBU - Stony Brook University [SUNY]
36 Vanderbilt University [Nashville]
37 Radboud University [Nijmegen]
38 University Health Network [Toronto, ON, Canada]
39 SRI - Sunnybrook Research Institute [Toronto]
40 UNSW - University of New South Wales [Sydney]
41 UZH - Universität Zürich [Zürich] = University of Zurich
42 Universiteit Utrecht / Utrecht University [Utrecht]
43 UNIGE - Université de Genève = University of Geneva
44 UKE - Universitaetsklinikum Hamburg-Eppendorf = University Medical Center Hamburg-Eppendorf [Hamburg]
45 Allen Institute for cell sciences
46 University of Warwick [Coventry]
47 UNIBE - Universität Bern / University of Bern
48 Simula Research Laboratory [OSLO]
49 NVIDIA - NVIDIA
50 Universität Heidelberg [Heidelberg] = Heidelberg University
51 UvA - University of Amsterdam [Amsterdam] = Universiteit van Amsterdam
52 NIH - National Institutes of Health [Bethesda, MD, USA]
53 TU Wien - Vienna University of Technology = Technische Universität Wien
54 University of Oulu
55 Edin. - University of Edinburgh
56 KU Leuven - Catholic University of Leuven = Katholieke Universiteit Leuven
57 SODA - Méthodes computationnelles et mathématiques pour comprendre la société et la santé à partir de données
58 PARIETAL - Modelling brain structure, function and variability based on high-field MRI data
59 NIAID-NIH - National Institute of Allergy and Infectious Diseases [Bethesda]
2 UCL - University College of London [London]
3 CONICET - Consejo Nacional de Investigaciones Científicas y Técnicas [Buenos Aires]
4 King‘s College London
5 Mila - Institut québécois d’intelligence artificielle
6 McGill University = Université McGill [Montréal, Canada]
7 Indiana University School of Medicine
8 HIT - Holon Institut of Technology
9 KU-ESAT - Department of Electrical Engineering [KU Leuven]
10 DKTK - German Cancer Consortium [Heidelberg]
11 ITU - IT University of Copenhagen
12 ISAS - Leibniz Institute for Analytical Sciences
13 Broad Institute [Cambridge]
14 University of Oxford
15 NCI-NIH - National Cancer Institute [Bethesda]
16 UPF - Universitat Pompeu Fabra [Barcelona]
17 Fraunhofer MEVIS - Fraunhofer Institute for Digital Medicine
18 Imperial College London
19 Leipzig University / Universität Leipzig
20 Perelman School of Medicine
21 University of Toronto
22 Radboud University Medical Center [Nijmegen]
23 LTSI - Laboratoire Traitement du Signal et de l'Image
24 Centre Hospitalier Universitaire de Rennes [CHU Rennes] = Rennes University Hospital [Ponchaillou]
25 University of Pennsylvania
26 University of Potsdam = Universität Potsdam
27 IHU Strasbourg - L'Institut hospitalo-universitaire de Strasbourg
28 Google Inc
29 Heidelberg University Hospital [Heidelberg]
30 AöR - University Hospital Essen
31 Helmholtz AI
32 SNU - Seoul National University [Seoul]
33 MUNI - Masaryk University [Brno]
34 EMBL Heidelberg
35 SBU - Stony Brook University [SUNY]
36 Vanderbilt University [Nashville]
37 Radboud University [Nijmegen]
38 University Health Network [Toronto, ON, Canada]
39 SRI - Sunnybrook Research Institute [Toronto]
40 UNSW - University of New South Wales [Sydney]
41 UZH - Universität Zürich [Zürich] = University of Zurich
42 Universiteit Utrecht / Utrecht University [Utrecht]
43 UNIGE - Université de Genève = University of Geneva
44 UKE - Universitaetsklinikum Hamburg-Eppendorf = University Medical Center Hamburg-Eppendorf [Hamburg]
45 Allen Institute for cell sciences
46 University of Warwick [Coventry]
47 UNIBE - Universität Bern / University of Bern
48 Simula Research Laboratory [OSLO]
49 NVIDIA - NVIDIA
50 Universität Heidelberg [Heidelberg] = Heidelberg University
51 UvA - University of Amsterdam [Amsterdam] = Universiteit van Amsterdam
52 NIH - National Institutes of Health [Bethesda, MD, USA]
53 TU Wien - Vienna University of Technology = Technische Universität Wien
54 University of Oulu
55 Edin. - University of Edinburgh
56 KU Leuven - Catholic University of Leuven = Katholieke Universiteit Leuven
57 SODA - Méthodes computationnelles et mathématiques pour comprendre la société et la santé à partir de données
58 PARIETAL - Modelling brain structure, function and variability based on high-field MRI data
59 NIAID-NIH - National Institute of Allergy and Infectious Diseases [Bethesda]
Annika Reinke
- Fonction : Auteur
- PersonId : 1356965
- ORCID : 0000-0003-4363-1876
Minu D. Tizabi
- Fonction : Auteur
- PersonId : 1357534
Pierre Jannin
- Fonction : Auteur
- PersonId : 740200
- IdHAL : pierre-jannin
- ORCID : 0000-0002-7415-071X
- IdRef : 116295848
Gaël Varoquaux
- Fonction : Auteur
- PersonId : 5878
- IdHAL : gael-varoquaux
- ORCID : 0000-0003-1076-5122
- IdRef : 126239894
Lena Maier-Hein
Connectez-vous pour contacter l'auteur
- Fonction : Auteur correspondant
- PersonId : 1122285
- ORCID : 0000-0003-4910-9368
Connectez-vous pour contacter l'auteur
Résumé
Validation metrics are key for tracking scientific progress and bridging the current chasm between artificial intelligence research and its translation into practice. However, increasing evidence shows that, particularly in image analysis, metrics are often chosen inadequately. Although taking into account the individual strengths, weaknesses and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multistage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides a reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Although focused on biomedical image analysis, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. The work serves to enhance global comprehension of a key topic in image analysis validation.