Search

Your search keyword '"Rädsch, Tim"' showing total 13 results
Start Over Author "Rädsch, Tim"
13 results on '"Rädsch, Tim"'

1. Metrics reloaded: recommendations for image analysis validation

Author

Maier-Hein, Lena, Reinke, Annika, Godau, Patrick, Tizabi, Minu D., Buettner, Florian, Christodoulou, Evangelia, Glocker, Ben, Isensee, Fabian, Kleesiek, Jens, Kozubek, Michal, Reyes, Mauricio, Riegler, Michael A., Wiesenfarth, Manuel, Kavur, A. Emre, Sudre, Carole H., Baumgartner, Michael, Eisenmann, Matthias, Heckmann-Nötzel, Doreen, Rädsch, Tim, Acion, Laura, Antonelli, Michela, Arbel, Tal, Bakas, Spyridon, Benis, Arriel, Blaschko, Matthew B., Cardoso, M. Jorge, Cheplygina, Veronika, Cimini, Beth A., Collins, Gary S., Farahani, Keyvan, Ferrer, Luciana, Galdran, Adrian, van Ginneken, Bram, Haase, Robert, Hashimoto, Daniel A., Hoffman, Michael M., Huisman, Merel, Jannin, Pierre, Kahn, Charles E., Kainmueller, Dagmar, Kainz, Bernhard, Karargyris, Alexandros, Karthikesalingam, Alan, Kofler, Florian, Kopp-Schneider, Annette, Kreshuk, Anna, Kurc, Tahsin, Landman, Bennett A., Litjens, Geert, Madani, Amin, Maier-Hein, Klaus, Martel, Anne L., Mattson, Peter, Meijering, Erik, Menze, Bjoern, Moons, Karel G. M., Müller, Henning, Nichyporuk, Brennan, Nickel, Felix, Petersen, Jens, Rajpoot, Nasir, Rieke, Nicola, Saez-Rodriguez, Julio, Sánchez, Clara I., Shetty, Shravya, van Smeden, Maarten, Summers, Ronald M., Taha, Abdel A., Tiulpin, Aleksei, Tsaftaris, Sotirios A., Van Calster, Ben, Varoquaux, Gaël, and Jäger, Paul F.

Published
2024
Full Text
View/download PDF

2. Understanding metric-related pitfalls in image analysis validation

Author

Reinke, Annika, Tizabi, Minu D., Baumgartner, Michael, Eisenmann, Matthias, Heckmann-Nötzel, Doreen, Kavur, A. Emre, Rädsch, Tim, Sudre, Carole H., Acion, Laura, Antonelli, Michela, Arbel, Tal, Bakas, Spyridon, Benis, Arriel, Buettner, Florian, Cardoso, M. Jorge, Cheplygina, Veronika, Chen, Jianxu, Christodoulou, Evangelia, Cimini, Beth A., Farahani, Keyvan, Ferrer, Luciana, Galdran, Adrian, van Ginneken, Bram, Glocker, Ben, Godau, Patrick, Hashimoto, Daniel A., Hoffman, Michael M., Huisman, Merel, Isensee, Fabian, Jannin, Pierre, Kahn, Charles E., Kainmueller, Dagmar, Kainz, Bernhard, Karargyris, Alexandros, Kleesiek, Jens, Kofler, Florian, Kooi, Thijs, Kopp-Schneider, Annette, Kozubek, Michal, Kreshuk, Anna, Kurc, Tahsin, Landman, Bennett A., Litjens, Geert, Madani, Amin, Maier-Hein, Klaus, Martel, Anne L., Meijering, Erik, Menze, Bjoern, Moons, Karel G. M., Müller, Henning, Nichyporuk, Brennan, Nickel, Felix, Petersen, Jens, Rafelski, Susanne M., Rajpoot, Nasir, Reyes, Mauricio, Riegler, Michael A., Rieke, Nicola, Saez-Rodriguez, Julio, Sánchez, Clara I., Shetty, Shravya, Summers, Ronald M., Taha, Abdel A., Tiulpin, Aleksei, Tsaftaris, Sotirios A., Van Calster, Ben, Varoquaux, Gaël, Yaniv, Ziv R., Jäger, Paul F., and Maier-Hein, Lena

Published
2024
Full Text
View/download PDF

4. Advancing Kidney, Kidney Tumor, Cyst Segmentation:A Multi-Planner U-Net Approach for the KiTS23 Challenge

Author

Heller, Nicholas, Wood, Andrew, Weight, Christopher, Isensee, Fabian, Rädsch, Tim, Teipaul, Resha, Papanikolopoulos, Nikolaos, Pandey, Sumit, Toshali, Perslev, Mathias, Dam, Erik B., Heller, Nicholas, Wood, Andrew, Weight, Christopher, Isensee, Fabian, Rädsch, Tim, Teipaul, Resha, Papanikolopoulos, Nikolaos, Pandey, Sumit, Toshali, Perslev, Mathias, and Dam, Erik B.

Abstract

Accurate segmentation of kidney tumors in medical images is crucial for effective treatment planning and patient outcomes prediction. The Kidney and Kidney Tumor Segmentation challenge (KiTS23) serves as a platform for evaluating advanced segmentation methods. In this study, we present our approach utilizing a Multi-Planner U-Net for kidney tumor segmentation. Our method combines the U-Net architecture with multiple image planes to enhance spatial information and improve segmentation accuracy. We employed a 3-fold cross-validation technique on the KiTS23 dataset, evaluating Mean Dice Score, precision, and recall metrics. Results indicate promising performance in segmenting Kidney + Tumor + Cyst and Tumor-only classes, while challenges persist in segmenting Tumor + Cyst cases. Our approach demonstrates potential in kidney tumor segmentation, with room for further refinement to address complex coexisting structures.

Published
2024

5. Why is the winner the best?

Author

Eisenmann, Matthias, Reinke, Annika, Weru, Vivienn, Tizabi, Minu Dietlinde, Isensee, Fabian, Adler, Tim J., Ali, Sharib, Andrearczyk, Vincent, Aubreville, Marc, Baid, Ujjwal, Bakas, Spyridon, Balu, Niranjan, Bano, Sophia, Bernal, Jorge, Bodenstedt, Sebastian, Casella, Alessandro, Cheplygina, Veronika, Daum, Marie, Bruijne, Marleen de, Depeursinge, Adrien, Dorent, Reuben, Egger, Jan, Ellis, David G., Engelhardt, Sandy, Ganz, Melanie, Ghatwary, Noha, Girard, Gabriel, Godau, Patrick, Gupta, Anubha, Hansen, Lasse, Harada, Kanako, Heinrich, Mattias, Heller, Nicholas, Hering, Alessa, Huaulmé, Arnaud, Jannin, Pierre, Kavur, Ali Emre, Kodym, Oldřich, Kozubek, Michal, Li, Jianning, Li, Hongwei, Ma, Jun, Martín-Isla, Carlos, Menze, Bjoern, Noble, Alison, Oreiller, Valentin, Padoy, Nicolas, Pati, Sarthak, Payette, Kelly, Rädsch, Tim, Rafael-Patiño, Jonathan, Bawa, Vivek Singh, Speidel, Stefanie, Sudre, Carole H., Wijnen, Kimberlin van, Wagner, Martin, Wei, Donglai, Yamlahi, Amine, Yap, Moi Hoon, Yuan, Chun, Zenk, Maximilian, Zia, Aneeq, Zimmerer, David, Aydogan, Dogu Baran, Bhattarai, Binod, Bloch, Louise, Brüngel, Raphael, Cho, Jihoon, Choi, Chanyeol, Dou, Qi, Ezhov, Ivan, Friedrich, Christoph M., Fuller, Clifton, Gaire, Rebati Raman, Galdran, Adrian, Faura, Álvaro García, Grammatikopoulou, Maria, Hong, SeulGi, Jahanifar, Mostafa, Jang, Ikbeom, Kadkhodamohammadi, Abdolrahim, Kang, Inha, Kofler, Florian, Kondo, Satoshi, Kuijf, Hugo, Li, Mingxing, Luu, Minh Huan, Martinčič, Tomaž, Morais, Pedro, Naser, Mohamed A., Oliveira, Bruno, Owen, David, Pang, Subeen, Park, Jinah, Park, Sung-Hong, Płotka, Szymon, Puybareau, Elodie, Rajpoot, Nasir, Ryu, Kanghyun, Saeed, Numan, Shephard, Adam, Shi, Pengcheng, Štepec, Dejan, Subedi, Ronast, Tochon, Guillaume, Torres, Helena R., Urien, Helene, Vilaça, João L., Wahid, Kareem Abdul, Wang, Haojie, Wang, Jiacheng, Wang, Liansheng, Wang, Xiyue, Wiestler, Benedikt, Wodzinski, Marek, Xia, Fangfang, Xie, Juanying, Xiong, Zhiwei, Yang, Sen, Yang, Yanwu, Zhao, Zixuan, Maier-Hein, Klaus, Jäger, Paul F., Kopp-Schneider, Annette, Maier-Hein, Lena, Eisenmann, Matthias, Reinke, Annika, Weru, Vivienn, Tizabi, Minu Dietlinde, Isensee, Fabian, Adler, Tim J., Ali, Sharib, Andrearczyk, Vincent, Aubreville, Marc, Baid, Ujjwal, Bakas, Spyridon, Balu, Niranjan, Bano, Sophia, Bernal, Jorge, Bodenstedt, Sebastian, Casella, Alessandro, Cheplygina, Veronika, Daum, Marie, Bruijne, Marleen de, Depeursinge, Adrien, Dorent, Reuben, Egger, Jan, Ellis, David G., Engelhardt, Sandy, Ganz, Melanie, Ghatwary, Noha, Girard, Gabriel, Godau, Patrick, Gupta, Anubha, Hansen, Lasse, Harada, Kanako, Heinrich, Mattias, Heller, Nicholas, Hering, Alessa, Huaulmé, Arnaud, Jannin, Pierre, Kavur, Ali Emre, Kodym, Oldřich, Kozubek, Michal, Li, Jianning, Li, Hongwei, Ma, Jun, Martín-Isla, Carlos, Menze, Bjoern, Noble, Alison, Oreiller, Valentin, Padoy, Nicolas, Pati, Sarthak, Payette, Kelly, Rädsch, Tim, Rafael-Patiño, Jonathan, Bawa, Vivek Singh, Speidel, Stefanie, Sudre, Carole H., Wijnen, Kimberlin van, Wagner, Martin, Wei, Donglai, Yamlahi, Amine, Yap, Moi Hoon, Yuan, Chun, Zenk, Maximilian, Zia, Aneeq, Zimmerer, David, Aydogan, Dogu Baran, Bhattarai, Binod, Bloch, Louise, Brüngel, Raphael, Cho, Jihoon, Choi, Chanyeol, Dou, Qi, Ezhov, Ivan, Friedrich, Christoph M., Fuller, Clifton, Gaire, Rebati Raman, Galdran, Adrian, Faura, Álvaro García, Grammatikopoulou, Maria, Hong, SeulGi, Jahanifar, Mostafa, Jang, Ikbeom, Kadkhodamohammadi, Abdolrahim, Kang, Inha, Kofler, Florian, Kondo, Satoshi, Kuijf, Hugo, Li, Mingxing, Luu, Minh Huan, Martinčič, Tomaž, Morais, Pedro, Naser, Mohamed A., Oliveira, Bruno, Owen, David, Pang, Subeen, Park, Jinah, Park, Sung-Hong, Płotka, Szymon, Puybareau, Elodie, Rajpoot, Nasir, Ryu, Kanghyun, Saeed, Numan, Shephard, Adam, Shi, Pengcheng, Štepec, Dejan, Subedi, Ronast, Tochon, Guillaume, Torres, Helena R., Urien, Helene, Vilaça, João L., Wahid, Kareem Abdul, Wang, Haojie, Wang, Jiacheng, Wang, Liansheng, Wang, Xiyue, Wiestler, Benedikt, Wodzinski, Marek, Xia, Fangfang, Xie, Juanying, Xiong, Zhiwei, Yang, Sen, Yang, Yanwu, Zhao, Zixuan, Maier-Hein, Klaus, Jäger, Paul F., Kopp-Schneider, Annette, and Maier-Hein, Lena

Abstract

International benchmarking competitions have become fundamental for the comparative performance assessment of image analysis methods. However, little attention has been given to investigating what can be learnt from these competitions. Do they really generate scientific progress? What are common and successful participation strategies? What makes a solution superior to a competing method? To address this gap in the literature, we performed a multi-center study with all 80 competitions that were conducted in the scope of IEEE ISBI 2021 and MICCAI 2021. Statistical analyses performed based on comprehensive descriptions of the submitted algorithms linked to their rank as well as the underlying participation strategies revealed common characteristics of winning solutions. These typically include the use of multi-task learning (63%) and/or multi-stage pipelines (61%), and a focus on augmentation (100%), image preprocessing (97%), data curation (79%), and postprocessing (66%). The "typical" lead of a winning team is a computer scientist with a doctoral degree, five years of experience in biomedical image analysis, and four years of experience in deep learning. Two core general development strategies stood out for highly-ranked teams: the reflection of the metrics in the method design and the focus on analyzing and handling failure cases. According to the organizers, 43% of the winning algorithms exceeded the state of the art but only 11% completely solved the respective domain problem. The insights of our study could help researchers (1) improve algorithm development strategies when approaching new problems, and (2) focus on open research questions revealed by this work.

Published
2023

6. Understanding metric-related pitfalls in image analysis validation

Author

Reinke, Annika, Tizabi, Minu D., Baumgartner, Michael, Eisenmann, Matthias, Heckmann-Nötzel, Doreen, Kavur, A. Emre, Rädsch, Tim, Sudre, Carole H., Acion, Laura, Antonelli, Michela, Arbel, Tal, Bakas, Spyridon, Benis, Arriel, Blaschko, Matthew, Buettner, Florian, Cardoso, M. Jorge, Cheplygina, Veronika, Chen, Jianxu, Christodoulou, Evangelia, Cimini, Beth A., Collins, Gary S., Farahani, Keyvan, Ferrer, Luciana, Galdran, Adrian, van Ginneken, Bram, Glocker, Ben, Godau, Patrick, Haase, Robert, Hashimoto, Daniel A., Hoffman, Michael M., Huisman, Merel, Isensee, Fabian, Jannin, Pierre, Kahn, Charles E., Kainmueller, Dagmar, Kainz, Bernhard, Karargyris, Alexandros, Karthikesalingam, Alan, Kenngott, Hannes, Kleesiek, Jens, Kofler, Florian, Kooi, Thijs, Kopp-Schneider, Annette, Kozubek, Michal, Kreshuk, Anna, Kurc, Tahsin, Landman, Bennett A., Litjens, Geert, Madani, Amin, Maier-Hein, Klaus, Martel, Anne L., Mattson, Peter, Meijering, Erik, Menze, Bjoern, Moons, Karel G. M., Müller, Henning, Nichyporuk, Brennan, Nickel, Felix, Petersen, Jens, Rafelski, Susanne M., Rajpoot, Nasir, Reyes, Mauricio, Riegler, Michael A., Rieke, Nicola, Saez-Rodriguez, Julio, Sánchez, Clara I., Shetty, Shravya, van Smeden, Maarten, Summers, Ronald M., Taha, Abdel A., Tiulpin, Aleksei, Tsaftaris, Sotirios A., Van Calster, Ben, Varoquaux, Gaël, Wiesenfarth, Manuel, Yaniv, Ziv R., Jäger, Paul F., Maier-Hein, Lena, Reinke, Annika, Tizabi, Minu D., Baumgartner, Michael, Eisenmann, Matthias, Heckmann-Nötzel, Doreen, Kavur, A. Emre, Rädsch, Tim, Sudre, Carole H., Acion, Laura, Antonelli, Michela, Arbel, Tal, Bakas, Spyridon, Benis, Arriel, Blaschko, Matthew, Buettner, Florian, Cardoso, M. Jorge, Cheplygina, Veronika, Chen, Jianxu, Christodoulou, Evangelia, Cimini, Beth A., Collins, Gary S., Farahani, Keyvan, Ferrer, Luciana, Galdran, Adrian, van Ginneken, Bram, Glocker, Ben, Godau, Patrick, Haase, Robert, Hashimoto, Daniel A., Hoffman, Michael M., Huisman, Merel, Isensee, Fabian, Jannin, Pierre, Kahn, Charles E., Kainmueller, Dagmar, Kainz, Bernhard, Karargyris, Alexandros, Karthikesalingam, Alan, Kenngott, Hannes, Kleesiek, Jens, Kofler, Florian, Kooi, Thijs, Kopp-Schneider, Annette, Kozubek, Michal, Kreshuk, Anna, Kurc, Tahsin, Landman, Bennett A., Litjens, Geert, Madani, Amin, Maier-Hein, Klaus, Martel, Anne L., Mattson, Peter, Meijering, Erik, Menze, Bjoern, Moons, Karel G. M., Müller, Henning, Nichyporuk, Brennan, Nickel, Felix, Petersen, Jens, Rafelski, Susanne M., Rajpoot, Nasir, Reyes, Mauricio, Riegler, Michael A., Rieke, Nicola, Saez-Rodriguez, Julio, Sánchez, Clara I., Shetty, Shravya, van Smeden, Maarten, Summers, Ronald M., Taha, Abdel A., Tiulpin, Aleksei, Tsaftaris, Sotirios A., Van Calster, Ben, Varoquaux, Gaël, Wiesenfarth, Manuel, Yaniv, Ziv R., Jäger, Paul F., and Maier-Hein, Lena

Abstract

Validation metrics are key for the reliable tracking of scientific progress and for bridging the current chasm between artificial intelligence (AI) research and its translation into practice. However, increasing evidence shows that particularly in image analysis, metrics are often chosen inadequately in relation to the underlying research problem. This could be attributed to a lack of accessibility of metric-related knowledge: While taking into account the individual strengths, weaknesses, and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multi-stage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides the first reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Focusing on biomedical image analysis but with the potential of transfer to other fields, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. To facilitate comprehension, illustrations and specific examples accompany each pitfall. As a structured body of information accessible to researchers of all levels of expertise, this work enhances global comprehension of a key topic in image analysis validation., Comment: Shared first authors: Annika Reinke and Minu D. Tizabi; shared senior authors: Lena Maier-Hein and Paul F. J\"ager. Published in Nature Methods. arXiv admin note: text overlap with arXiv:2206.01653

Published
2023
Full Text
View/download PDF

7. Labeling instructions matter in biomedical image analysis

Author

Rädsch, Tim, Reinke, Annika, Weru, Vivienn, Tizabi, Minu D., Schreck, Nicholas, Kavur, A. Emre, Pekdemir, Bünyamin, Roß, Tobias, Kopp-Schneider, Annette, Maier-Hein, Lena, Rädsch, Tim, Reinke, Annika, Weru, Vivienn, Tizabi, Minu D., Schreck, Nicholas, Kavur, A. Emre, Pekdemir, Bünyamin, Roß, Tobias, Kopp-Schneider, Annette, and Maier-Hein, Lena

Abstract

Biomedical image analysis algorithm validation depends on high-quality annotation of reference datasets, for which labeling instructions are key. Despite their importance, their optimization remains largely unexplored. Here, we present the first systematic study of labeling instructions and their impact on annotation quality in the field. Through comprehensive examination of professional practice and international competitions registered at the MICCAI Society, we uncovered a discrepancy between annotators' needs for labeling instructions and their current quality and availability. Based on an analysis of 14,040 images annotated by 156 annotators from four professional companies and 708 Amazon Mechanical Turk (MTurk) crowdworkers using instructions with different information density levels, we further found that including exemplary images significantly boosts annotation performance compared to text-only descriptions, while solely extending text descriptions does not. Finally, professional annotators constantly outperform MTurk crowdworkers. Our study raises awareness for the need of quality standards in biomedical image analysis labeling instructions.

Published
2022

8. Metrics reloaded: Recommendations for image analysis validation

Author

Maier-Hein, Lena, Reinke, Annika, Godau, Patrick, Tizabi, Minu D., Buettner, Florian, Christodoulou, Evangelia, Glocker, Ben, Isensee, Fabian, Kleesiek, Jens, Kozubek, Michal, Reyes, Mauricio, Riegler, Michael A., Wiesenfarth, Manuel, Kavur, A. Emre, Sudre, Carole H., Baumgartner, Michael, Eisenmann, Matthias, Heckmann-Nötzel, Doreen, Rädsch, Tim, Acion, Laura, Antonelli, Michela, Arbel, Tal, Bakas, Spyridon, Benis, Arriel, Blaschko, Matthew, Cardoso, M. Jorge, Cheplygina, Veronika, Cimini, Beth A., Collins, Gary S., Farahani, Keyvan, Ferrer, Luciana, Galdran, Adrian, van Ginneken, Bram, Haase, Robert, Hashimoto, Daniel A., Hoffman, Michael M., Huisman, Merel, Jannin, Pierre, Kahn, Charles E., Kainmueller, Dagmar, Kainz, Bernhard, Karargyris, Alexandros, Karthikesalingam, Alan, Kenngott, Hannes, Kofler, Florian, Kopp-Schneider, Annette, Kreshuk, Anna, Kurc, Tahsin, Landman, Bennett A., Litjens, Geert, Madani, Amin, Maier-Hein, Klaus, Martel, Anne L., Mattson, Peter, Meijering, Erik, Menze, Bjoern, Moons, Karel G. M., Müller, Henning, Nichyporuk, Brennan, Nickel, Felix, Petersen, Jens, Rajpoot, Nasir, Rieke, Nicola, Saez-Rodriguez, Julio, Sánchez, Clara I., Shetty, Shravya, van Smeden, Maarten, Summers, Ronald M., Taha, Abdel A., Tiulpin, Aleksei, Tsaftaris, Sotirios A., Van Calster, Ben, Varoquaux, Gaël, Jäger, Paul F., Maier-Hein, Lena, Reinke, Annika, Godau, Patrick, Tizabi, Minu D., Buettner, Florian, Christodoulou, Evangelia, Glocker, Ben, Isensee, Fabian, Kleesiek, Jens, Kozubek, Michal, Reyes, Mauricio, Riegler, Michael A., Wiesenfarth, Manuel, Kavur, A. Emre, Sudre, Carole H., Baumgartner, Michael, Eisenmann, Matthias, Heckmann-Nötzel, Doreen, Rädsch, Tim, Acion, Laura, Antonelli, Michela, Arbel, Tal, Bakas, Spyridon, Benis, Arriel, Blaschko, Matthew, Cardoso, M. Jorge, Cheplygina, Veronika, Cimini, Beth A., Collins, Gary S., Farahani, Keyvan, Ferrer, Luciana, Galdran, Adrian, van Ginneken, Bram, Haase, Robert, Hashimoto, Daniel A., Hoffman, Michael M., Huisman, Merel, Jannin, Pierre, Kahn, Charles E., Kainmueller, Dagmar, Kainz, Bernhard, Karargyris, Alexandros, Karthikesalingam, Alan, Kenngott, Hannes, Kofler, Florian, Kopp-Schneider, Annette, Kreshuk, Anna, Kurc, Tahsin, Landman, Bennett A., Litjens, Geert, Madani, Amin, Maier-Hein, Klaus, Martel, Anne L., Mattson, Peter, Meijering, Erik, Menze, Bjoern, Moons, Karel G. M., Müller, Henning, Nichyporuk, Brennan, Nickel, Felix, Petersen, Jens, Rajpoot, Nasir, Rieke, Nicola, Saez-Rodriguez, Julio, Sánchez, Clara I., Shetty, Shravya, van Smeden, Maarten, Summers, Ronald M., Taha, Abdel A., Tiulpin, Aleksei, Tsaftaris, Sotirios A., Van Calster, Ben, Varoquaux, Gaël, and Jäger, Paul F.

Abstract

Increasing evidence shows that flaws in machine learning (ML) algorithm validation are an underestimated global problem. Particularly in automatic biomedical image analysis, chosen performance metrics often do not reflect the domain interest, thus failing to adequately measure scientific progress and hindering translation of ML techniques into practice. To overcome this, our large international expert consortium created Metrics Reloaded, a comprehensive framework guiding researchers in the problem-aware selection of metrics. Following the convergence of ML methodology across application domains, Metrics Reloaded fosters the convergence of validation methodology. The framework was developed in a multi-stage Delphi process and is based on the novel concept of a problem fingerprint - a structured representation of the given problem that captures all aspects that are relevant for metric selection, from the domain interest to the properties of the target structure(s), data set and algorithm output. Based on the problem fingerprint, users are guided through the process of choosing and applying appropriate validation metrics while being made aware of potential pitfalls. Metrics Reloaded targets image analysis problems that can be interpreted as a classification task at image, object or pixel level, namely image-level classification, object detection, semantic segmentation, and instance segmentation tasks. To improve the user experience, we implemented the framework in the Metrics Reloaded online tool, which also provides a point of access to explore weaknesses, strengths and specific recommendations for the most common validation metrics. The broad applicability of our framework across domains is demonstrated by an instantiation for various biological and medical image analysis use cases., Comment: Shared first authors: Lena Maier-Hein, Annika Reinke. arXiv admin note: substantial text overlap with arXiv:2104.05642 Published in Nature Methods

Published
2022
Full Text
View/download PDF

9. Common Limitations of Image Processing Metrics: A Picture Story

Author

Reinke, Annika, Tizabi, Minu D., Sudre, Carole H., Eisenmann, Matthias, Rädsch, Tim, Baumgartner, Michael, Acion, Laura, Antonelli, Michela, Arbel, Tal, Bakas, Spyridon, Bankhead, Peter, Benis, Arriel, Cardoso, M. Jorge, Cheplygina, Veronika, Christodoulou, Evangelia, Cimini, Beth, Collins, Gary S., Farahani, Keyvan, van Ginneken, Bram, Glocker, Ben, Godau, Patrick, Hamprecht, Fred, Hashimoto, Daniel A., Heckmann-Nötzel, Doreen, Hoffman, Michael M., Huisman, Merel, Isensee, Fabian, Jannin, Pierre, Kahn, Charles E., Karargyris, Alexandros, Karthikesalingam, Alan, Kainz, Bernhard, Kavur, Emre, Kenngott, Hannes, Kleesiek, Jens, Kooi, Thijs, Kozubek, Michal, Kreshuk, Anna, Kurc, Tahsin, Landman, Bennett A., Litjens, Geert, Madani, Amin, Maier-Hein, Klaus, Martel, Anne L., Mattson, Peter, Meijering, Erik, Menze, Bjoern, Moher, David, Moons, Karel G. M., Müller, Henning, Nichyporuk, Brennan, Nickel, Felix, Noyan, M. Alican, Petersen, Jens, Polat, Gorkem, Rajpoot, Nasir, Reyes, Mauricio, Rieke, Nicola, Riegler, Michael, Rivaz, Hassan, Saez-Rodriguez, Julio, Gutierrez, Clarisa Sanchez, Schroeter, Julien, Saha, Anindo, Shetty, Shravya, van Smeden, Maarten, Stieltjes, Bram, Summers, Ronald M., Taha, Abdel A., Tsaftaris, Sotirios A., Van Calster, Ben, Varoquaux, Gaël, Wiesenfarth, Manuel, Yaniv, Ziv R., Kopp-Schneider, Annette, Jäger, Paul, and Maier-Hein, Lena

Subjects
FOS: Computer and information sciences, Computer Vision and Pattern Recognition (cs.CV), Image and Video Processing (eess.IV), FOS: Electrical engineering, electronic engineering, information engineering, Computer Science - Computer Vision and Pattern Recognition, Electrical Engineering and Systems Science - Image and Video Processing
Abstract

While the importance of automatic image analysis is continuously increasing, recent meta-research revealed major flaws with respect to algorithm validation. Performance metrics are particularly key for meaningful, objective, and transparent performance assessment and validation of the used automatic algorithms, but relatively little attention has been given to the practical pitfalls when using specific metrics for a given image analysis task. These are typically related to (1) the disregard of inherent metric properties, such as the behaviour in the presence of class imbalance or small target structures, (2) the disregard of inherent data set properties, such as the non-independence of the test cases, and (3) the disregard of the actual biomedical domain interest that the metrics should reflect. This living dynamically document has the purpose to illustrate important limitations of performance metrics commonly applied in the field of image analysis. In this context, it focuses on biomedical image analysis problems that can be phrased as image-level classification, semantic segmentation, instance segmentation, or object detection task. The current version is based on a Delphi process on metrics conducted by an international consortium of image analysis experts from more than 60 institutions worldwide., This is a dynamic paper on limitations of commonly used metrics. The current version discusses metrics for image-level classification, semantic segmentation, object detection and instance segmentation. For missing use cases, comments or questions, please contact a.reinke@dkfz.de or l.maier-hein@dkfz.de. Substantial contributions to this document will be acknowledged with a co-authorship

Published
2021

10. Common Limitations of Image Processing Metrics:A Picture Story

Author

Reinke, Annika, Tizabi, Minu D., Sudre, Carole H., Eisenmann, Matthias, Rädsch, Tim, Baumgartner, Michael, Acion, Laura, Antonelli, Michela, Bakas, Spyridon, Bankhead, Peter, Benis, Arriel, Cardoso, M. Jorge, Cheplygina, Veronika, Christodoulou, Evangelia, Cimini, Beth, Collins, Gary S., Farahani, Keyvan, Ginneken, Bram van, Glocker, Ben, Godau, Patrick, Hamprecht, Fred, Hashimoto, Daniel A., Heckmann-Nötzel, Doreen, Hoffman, Michael M., Huisman, Merel, Isensee, Fabian, Jannin, Pierre, Kahn, Charles E., Karargyris, Alexandros, Karthikesalingam, Alan, Kainz, Bernhard, Kavur, Emre, Kenngott, Hannes, Kleesiek, Jens, Kooi, Thijs, Kozubek, Michal, Kreshuk, Anna, Kurc, Tahsin, Landman, Bennett A., Litjens, Geert, Madani, Amin, Maier-Hein, Klaus, Martel, Anne L., Mattson, Peter, Meijering, Erik, Menze, Bjoern, Moher, David, Moons, Karel G. M., Müller, Henning, Nichyporuk, Brennan, Nickel, Felix, Noyan, M. Alican, Petersen, Jens, Polat, Gorkem, Rajpoot, Nasir, Reyes, Mauricio, Rieke, Nicola, Riegler, Michael, Rivaz, Hassan, Saez-Rodriguez, Julio, Gutierrez, Clarisa Sanchez, Schroeter, Julien, Saha, Anindo, Shetty, Shravya, Smeden, Maarten van, Stieltjes, Bram, Summers, Ronald M., Taha, Abdel A., Tsaftaris, Sotirios A., Calster, Ben Van, Varoquaux, Gaël, Wiesenfarth, Manuel, Yaniv, Ziv R., Kopp-Schneider, Annette, Jäger, Paul, and Maier-Hein, Lena

Abstract

While the importance of automatic image analysis is continuously increasing, recent meta-research revealed major flaws with respect to algorithm validation. Performance metrics are particularly key for meaningful, objective, and transparent performance assessment and validation of the used automatic algorithms, but relatively little attention has been given to the practical pitfalls when using specific metrics for a given image analysis task. These are typically related to (1) the disregard of inherent metric properties, such as the behaviour in the presence of class imbalance or small target structures, (2) the disregard of inherent data set properties, such as the non-independence of the test cases, and (3) the disregard of the actual biomedical domain interest that the metrics should reflect. This living dynamically document has the purpose to illustrate important limitations of performance metrics commonly applied in the field of image analysis. In this context, it focuses on biomedical image analysis problems that can be phrased as image-level classification, semantic segmentation, instance segmentation, or object detection task. The current version is based on a Delphi process on metrics conducted by an international consortium of image analysis experts from more than 60 institutions worldwide.

Published
2021

13. Understanding metric-related pitfalls in image analysis validation.

Author

Reinke A, Tizabi MD, Baumgartner M, Eisenmann M, Heckmann-Nötzel D, Kavur AE, Rädsch T, Sudre CH, Acion L, Antonelli M, Arbel T, Bakas S, Benis A, Blaschko M, Buettner F, Cardoso MJ, Cheplygina V, Chen J, Christodoulou E, Cimini BA, Collins GS, Farahani K, Ferrer L, Galdran A, van Ginneken B, Glocker B, Godau P, Haase R, Hashimoto DA, Hoffman MM, Huisman M, Isensee F, Jannin P, Kahn CE, Kainmueller D, Kainz B, Karargyris A, Karthikesalingam A, Kenngott H, Kleesiek J, Kofler F, Kooi T, Kopp-Schneider A, Kozubek M, Kreshuk A, Kurc T, Landman BA, Litjens G, Madani A, Maier-Hein K, Martel AL, Mattson P, Meijering E, Menze B, Moons KGM, Müller H, Nichyporuk B, Nickel F, Petersen J, Rafelski SM, Rajpoot N, Reyes M, Riegler MA, Rieke N, Saez-Rodriguez J, Sánchez CI, Shetty S, van Smeden M, Summers RM, Taha AA, Tiulpin A, Tsaftaris SA, Calster BV, Varoquaux G, Wiesenfarth M, Yaniv ZR, Jäger PF, and Maier-Hein L

Abstract

Validation metrics are key for the reliable tracking of scientific progress and for bridging the current chasm between artificial intelligence (AI) research and its translation into practice. However, increasing evidence shows that particularly in image analysis, metrics are often chosen inadequately in relation to the underlying research problem. This could be attributed to a lack of accessibility of metric-related knowledge: While taking into account the individual strengths, weaknesses, and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multi-stage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides the first reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Focusing on biomedical image analysis but with the potential of transfer to other fields, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. To facilitate comprehension, illustrations and specific examples accompany each pitfall. As a structured body of information accessible to researchers of all levels of expertise, this work enhances global comprehension of a key topic in image analysis validation.

Published
2024

Catalog

Books, media, physical & digital resources
See catalog results