Search

Your search keyword '"Annika Reinke"' showing total 64 results
Start Over Author "Annika Reinke"
64 results on '"Annika Reinke"'

1. Radiomics workflow definition & challenges - German priority program 2177 consensus statement on clinically applied radiomics

Author

Ralf Floca, Jonas Bohn, Christian Haux, Benedikt Wiestler, Frank G. Zöllner, Annika Reinke, Jakob Weiß, Marco Nolden, Steffen Albert, Thorsten Persigehl, Tobias Norajitra, Bettina Baeßler, Marc Dewey, Rickmer Braren, Martin Büchert, Eva Maria Fallenberg, Norbert Galldiks, Annika Gerken, Michael Götz, Horst K. Hahn, Johannes Haubold, Tobias Haueise, Nils Große Hokamp, Michael Ingrisch, Andra-Iza Iuga, Marco Janoschke, Matthias Jung, Lena Sophie Kiefer, Philipp Lohmann, Jürgen Machann, Jan Hendrik Moltz, Johanna Nattenmüller, Tobias Nonnenmacher, Benedict Oerther, Ahmed E. Othman, Felix Peisen, Fritz Schick, Lale Umutlu, Barbara D. Wichtmann, Wenzhao Zhao, Svenja Caspers, Heinz-Peter Schlemmer, Christopher L. Schlett, Klaus Maier-Hein, and Fabian Bamberg

Subjects
Image processing, Computer-assisted, Workflow, Terminology, Consensus development conference, Medical physics. Medical radiology. Nuclear medicine, R895-920
Abstract

Abstract Objectives Achieving a consensus on a definition for different aspects of radiomics workflows to support their translation into clinical usage. Furthermore, to assess the perspective of experts on important challenges for a successful clinical workflow implementation. Materials and methods The consensus was achieved by a multi-stage process. Stage 1 comprised a definition screening, a retrospective analysis with semantic mapping of terms found in 22 workflow definitions, and the compilation of an initial baseline definition. Stages 2 and 3 consisted of a Delphi process with over 45 experts hailing from sites participating in the German Research Foundation (DFG) Priority Program 2177. Stage 2 aimed to achieve a broad consensus for a definition proposal, while stage 3 identified the importance of translational challenges. Results Workflow definitions from 22 publications (published 2012–2020) were analyzed. Sixty-nine definition terms were extracted, mapped, and semantic ambiguities (e.g., homonymous and synonymous terms) were identified and resolved. The consensus definition was developed via a Delphi process. The final definition comprising seven phases and 37 aspects reached a high overall consensus (> 89% of experts “agree” or “strongly agree”). Two aspects reached no strong consensus. In addition, the Delphi process identified and characterized from the participating experts’ perspective the ten most important challenges in radiomics workflows. Conclusion To overcome semantic inconsistencies between existing definitions and offer a well-defined, broad, referenceable terminology, a consensus workflow definition for radiomics-based setups and a terms mapping to existing literature was compiled. Moreover, the most relevant challenges towards clinical application were characterized. Critical relevance statement Lack of standardization represents one major obstacle to successful clinical translation of radiomics. Here, we report a consensus workflow definition on different aspects of radiomics studies and highlight important challenges to advance the clinical adoption of radiomics. Key Points Published radiomics workflow terminologies are inconsistent, hindering standardization and translation. A consensus radiomics workflow definition proposal with high agreement was developed. Publicly available result resources for further exploitation by the scientific community. Graphical Abstract

Published
2024
Full Text
View/download PDF

2. The Medical Segmentation Decathlon

Author

Michela Antonelli, Annika Reinke, Spyridon Bakas, Keyvan Farahani, Annette Kopp-Schneider, Bennett A. Landman, Geert Litjens, Bjoern Menze, Olaf Ronneberger, Ronald M. Summers, Bram van Ginneken, Michel Bilello, Patrick Bilic, Patrick F. Christ, Richard K. G. Do, Marc J. Gollub, Stephan H. Heckers, Henkjan Huisman, William R. Jarnagin, Maureen K. McHugo, Sandy Napel, Jennifer S. Golia Pernicka, Kawal Rhode, Catalina Tobon-Gomez, Eugene Vorontsov, James A. Meakin, Sebastien Ourselin, Manuel Wiesenfarth, Pablo Arbeláez, Byeonguk Bae, Sihong Chen, Laura Daza, Jianjiang Feng, Baochun He, Fabian Isensee, Yuanfeng Ji, Fucang Jia, Ildoo Kim, Klaus Maier-Hein, Dorit Merhof, Akshay Pai, Beomhee Park, Mathias Perslev, Ramin Rezaiifar, Oliver Rippel, Ignacio Sarasua, Wei Shen, Jaemin Son, Christian Wachinger, Liansheng Wang, Yan Wang, Yingda Xia, Daguang Xu, Zhanwei Xu, Yefeng Zheng, Amber L. Simpson, Lena Maier-Hein, and M. Jorge Cardoso

Subjects
Science
Abstract

International challenges have become the de facto standard for comparative assessment of image analysis algorithms. Here, the authors present the results of a biomedical image segmentation challenge, showing that a method capable of performing well on multiple tasks will generalize well to a previously unseen task.

Published
2022
Full Text
View/download PDF

3. Heidelberg colorectal data set for surgical data science in the sensor operating room

Author

Lena Maier-Hein, Martin Wagner, Tobias Ross, Annika Reinke, Sebastian Bodenstedt, Peter M. Full, Hellena Hempe, Diana Mindroc-Filimon, Patrick Scholz, Thuy Nuong Tran, Pierangela Bruno, Anna Kisilenko, Benjamin Müller, Tornike Davitashvili, Manuela Capek, Minu D. Tizabi, Matthias Eisenmann, Tim J. Adler, Janek Gröhl, Melanie Schellenberg, Silvia Seidlitz, T. Y. Emmy Lai, Bünyamin Pekdemir, Veith Roethlingshoefer, Fabian Both, Sebastian Bittel, Marc Mengler, Lars Mündermann, Martin Apitz, Annette Kopp-Schneider, Stefanie Speidel, Felix Nickel, Pascal Probst, Hannes G. Kenngott, and Beat P. Müller-Stich

Subjects
Science
Abstract

Measurement(s) colorectum Technology Type(s) Laparoscopy Factor Type(s) surgery type Sample Characteristic - Organism Homo sapiens Machine-accessible metadata file describing the reported data: https://doi.org/10.6084/m9.figshare.14178773

Published
2021
Full Text
View/download PDF

4. Ten years of image analysis and machine learning competitions in dementia

Author

Esther E. Bron, Stefan Klein, Annika Reinke, Janne M. Papma, Lena Maier-Hein, Daniel C. Alexander, and Neil P. Oxtoby

Subjects
Grand challenges, Alzheimer’s disease, Neuroimaging, Machine learning, Validation, Neurosciences. Biological psychiatry. Neuropsychiatry, RC321-571
Abstract

Machine learning methods exploiting multi-parametric biomarkers, especially based on neuroimaging, have huge potential to improve early diagnosis of dementia and to predict which individuals are at-risk of developing dementia. To benchmark algorithms in the field of machine learning and neuroimaging in dementia and assess their potential for use in clinical practice and clinical trials, seven grand challenges have been organized in the last decade: MIRIAD (2012), Alzheimer’s Disease Big Data DREAM (2014), CADDementia (2014), Machine Learning Challenge (2014), MCI Neuroimaging (2017), TADPOLE (2017), and the Predictive Analytics Competition (2019). Based on two challenge evaluation frameworks, we analyzed how these grand challenges are complementing each other regarding research questions, datasets, validation approaches, results and impact.The seven grand challenges addressed questions related to screening, clinical status estimation, prediction and monitoring in (pre-clinical) dementia. There was little overlap in clinical questions, tasks and performance metrics. Whereas this aids providing insight on a broad range of questions, it also limits the validation of results across challenges. The validation process itself was mostly comparable between challenges, using similar methods for ensuring objective comparison, uncertainty estimation and statistical testing. In general, winning algorithms performed rigorous data pre-processing and combined a wide range of input features.Despite high state-of-the-art performances, most of the methods evaluated by the challenges are not clinically used. To increase impact, future challenges could pay more attention to statistical analysis of which factors (i.e., features, models) relate to higher performance, to clinical questions beyond Alzheimer’s disease, and to using testing data beyond the Alzheimer’s Disease Neuroimaging Initiative. Grand challenges would be an ideal venue for assessing the generalizability of algorithm performance to unseen data of other cohorts. Key for increasing impact in this way are larger testing data sizes, which could be reached by sharing algorithms rather than data to exploit data that cannot be shared. Given the potential and lessons learned in the past ten years, we are excited by the prospects of grand challenges in machine learning and neuroimaging for the next ten years and beyond.

Published
2022
Full Text
View/download PDF

5. Semantic segmentation of multispectral photoacoustic images using deep learning

Author

Melanie Schellenberg, Kris K. Dreher, Niklas Holzwarth, Fabian Isensee, Annika Reinke, Nicholas Schreck, Alexander Seitel, Minu D. Tizabi, Lena Maier-Hein, and Janek Gröhl

Subjects
Medical image segmentation, Deep learning, Multispectral imaging, Photoacoustics, Optoacoustics, Physics, QC1-999, Acoustics. Sound, QC221-246, Optics. Light, QC350-467
Abstract

Photoacoustic (PA) imaging has the potential to revolutionize functional medical imaging in healthcare due to the valuable information on tissue physiology contained in multispectral photoacoustic measurements. Clinical translation of the technology requires conversion of the high-dimensional acquired data into clinically relevant and interpretable information. In this work, we present a deep learning-based approach to semantic segmentation of multispectral photoacoustic images to facilitate image interpretability. Manually annotated photoacoustic and ultrasound imaging data are used as reference and enable the training of a deep learning-based segmentation algorithm in a supervised manner. Based on a validation study with experimentally acquired data from 16 healthy human volunteers, we show that automatic tissue segmentation can be used to create powerful analyses and visualizations of multispectral photoacoustic images. Due to the intuitive representation of high-dimensional information, such a preprocessing algorithm could be a valuable means to facilitate the clinical translation of photoacoustic imaging.

Published
2022
Full Text
View/download PDF

6. Methods and open-source toolkit for analyzing and visualizing challenge results

Author

Manuel Wiesenfarth, Annika Reinke, Bennett A. Landman, Matthias Eisenmann, Laura Aguilera Saiz, M. Jorge Cardoso, Lena Maier-Hein, and Annette Kopp-Schneider

Subjects
Medicine, Science
Abstract

Abstract Grand challenges have become the de facto standard for benchmarking image analysis algorithms. While the number of these international competitions is steadily increasing, surprisingly little effort has been invested in ensuring high quality design, execution and reporting for these international competitions. Specifically, results analysis and visualization in the event of uncertainties have been given almost no attention in the literature. Given these shortcomings, the contribution of this paper is two-fold: (1) we present a set of methods to comprehensively analyze and visualize the results of single-task and multi-task challenges and apply them to a number of simulated and real-life challenges to demonstrate their specific strengths and weaknesses; (2) we release the open-source framework challengeR as part of this work to enable fast and wide adoption of the methodology proposed in this paper. Our approach offers an intuitive way to gain important insights into the relative and absolute performance of algorithms, which cannot be revealed by commonly applied visualization techniques. This is demonstrated by the experiments performed in the specific context of biomedical image analysis challenges. Our framework could thus become an important tool for analyzing and visualizing challenge results in the field of biomedical image analysis and beyond.

Published
2021
Full Text
View/download PDF

8. Why rankings of biomedical image analysis competitions should be interpreted with care

Author

Lena Maier-Hein, Matthias Eisenmann, Annika Reinke, Sinan Onogur, Marko Stankovic, Patrick Scholz, Tal Arbel, Hrvoje Bogunovic, Andrew P. Bradley, Aaron Carass, Carolin Feldmann, Alejandro F. Frangi, Peter M. Full, Bram van Ginneken, Allan Hanbury, Katrin Honauer, Michal Kozubek, Bennett A. Landman, Keno März, Oskar Maier, Klaus Maier-Hein, Bjoern H. Menze, Henning Müller, Peter F. Neher, Wiro Niessen, Nasir Rajpoot, Gregory C. Sharp, Korsuk Sirinukunwattana, Stefanie Speidel, Christian Stock, Danail Stoyanov, Abdel Aziz Taha, Fons van der Sommen, Ching-Wei Wang, Marc-André Weber, Guoyan Zheng, Pierre Jannin, and Annette Kopp-Schneider

Subjects
Science
Abstract

Biomedical image analysis challenges have increased in the last ten years, but common practices have not been established yet. Here the authors analyze 150 recent challenges and demonstrate that outcome varies based on the metrics used and that limited information reporting hampers reproducibility.

Published
2018
Full Text
View/download PDF

10. Author Correction: Why rankings of biomedical image analysis competitions should be interpreted with care

Author

Lena Maier-Hein, Matthias Eisenmann, Annika Reinke, Sinan Onogur, Marko Stankovic, Patrick Scholz, Tal Arbel, Hrvoje Bogunovic, Andrew P. Bradley, Aaron Carass, Carolin Feldmann, Alejandro F. Frangi, Peter M. Full, Bram van Ginneken, Allan Hanbury, Katrin Honauer, Michal Kozubek, Bennett A. Landman, Keno März, Oskar Maier, Klaus Maier-Hein, Bjoern H. Menze, Henning Müller, Peter F. Neher, Wiro Niessen, Nasir Rajpoot, Gregory C. Sharp, Korsuk Sirinukunwattana, Stefanie Speidel, Christian Stock, Danail Stoyanov, Abdel Aziz Taha, Fons van der Sommen, Ching-Wei Wang, Marc-André Weber, Guoyan Zheng, Pierre Jannin, and Annette Kopp-Schneider

Subjects
Science
Abstract

In the original version of this Article the values in the rightmost column of Table 1 were inadvertently shifted relative to the other columns. This has now been corrected in the PDF and HTML versions of the Article.

Published
2019
Full Text
View/download PDF

13. Why is the Winner the Best?

Author

Matthias Eisenmann, Annika Reinke, Vivienn Weru, Minu Dietlinde Tizabi, Fabian Isensee, Tim J. Adler, Sharib Ali, Vincent Andrearczyk, Marc Aubreville, Ujjwal Baid, Spyridon Bakas, Niranjan Balu, Sophia Bano, Jorge Bernal, Sebastian Bodenstedt, Alessandro Casella, Veronika Cheplygina, Marie Daum, Marleen de Bruijne, Adrien Depeursinge, Reuben Dorent, Jan Egger, David G. Ellis, Sandy Engelhardt, Melanie Ganz, Noha M. Ghatwary, Gabriel Girard, Patrick Godau, Anubha Gupta, Lasse Hansen, Kanako Harada, Mattias P. Heinrich, Nicholas Heller, Alessa Hering, Arnaud Huaulmé, Pierre Jannin, A. Emre Kavur, Oldrich Kodym, Michal Kozubek 0001, Jianning Li, Hongwei Bran Li, Jun Ma 0016, Carlos Martín-Isla, Bjoern H. Menze, J. Alison Noble, Valentin Oreiller, Nicolas Padoy, Sarthak Pati, Kelly Payette, Tim Rädsch, Jonathan Rafael-Patino, Vivek Singh Bawa, Stefanie Speidel, Carole H. Sudre, Kimberlin M. H. van Wijnen, Martin Wagner 0001, D. Wei, Amine Yamlahi, Moi Hoon Yap, C. Yuan, Maximilian Zenk, A. Zia, David Zimmerer, Dogu Baran Aydogan, Binod Bhattarai, Louise Bloch, Raphael Brüngel, J. Cho, C. Choi, Q. Dou, Ivan Ezhov, Christoph M. Friedrich, C. Fuller, Rebati Raman Gaire, Adrian Galdran, álvaro García-Faura, Maria Grammatikopoulou, S. Hong, Mostafa Jahanifar, I. Jang, Abdolrahim Kadkhodamohammadi, I. Kang, Florian Kofler, S. Kondo, Hugo Jaco Kuijf, M. Li, M. Luu, Tomaz Martincic, Pedro Morais, Mohamed A. Naser, Bruno Oliveira 0002, David Owen 0001, S. Pang, J. Park, S. Park, Szymon Plotka, élodie Puybareau, Nasir M. Rajpoot, K. Ryu, Numan Saeed, Adam Shephard, Pengcheng Shi, Dejan Stepec, Ronast Subedi, Guillaume Tochon, Helena R. Torres, Hélène Urien, João L. Vilaça, Kareem A. Wahid, H. Wang, J. Wang, L. Wang, X. Wang, Benedikt Wiestler, Marek Wodzinski, F. Xia, J. Xie, Z. Xiong, S. Yang, Y. Yang, Z. Zhao, Klaus H. Maier-Hein, Paul F. Jäger, Annette Kopp-Schneider, and Lena Maier-Hein

Published
2023
Full Text
View/download PDF

19. Understanding metric-related pitfalls in image analysis validation.

Author

Annika Reinke, Minu Tizabi, Michael Baumgartner 0001, Matthias Eisenmann, Doreen Heckmann-Nötzel, A. Emre Kavur, Tim Rädsch, Carole H. Sudre, Laura Ación, Michela Antonelli, Tal Arbel, Spyridon Bakas, Arriel Benis, Matthew B. Blaschko, Florian Büttner 0001, M. Jorge Cardoso, Veronika Cheplygina, Jianxu Chen 0001, Evangelia Christodoulou, Beth A. Cimini, Gary S. Collins, Keyvan Farahani, Luciana Ferrer, Adrian Galdran, Bram van Ginneken, Ben Glocker, Patrick Godau, Robert Haase, Daniel A. Hashimoto, Michael M. Hoffman, Merel Huisman, Fabian Isensee, Pierre Jannin, Charles E. Kahn, Dagmar Kainmueller, Bernhard Kainz, Alexandros Karargyris, Alan Karthikesalingam, Hannes Kenngott, Jens Kleesiek, Florian Kofler, Thijs Kooi, Annette Kopp-Schneider, Michal Kozubek 0001, Anna Kreshuk, Tahsin M. Kurç, Bennett A. Landman, Geert Litjens 0001, Amin Madani, Klaus H. Maier-Hein, Anne L. Martel, Peter Mattson, Erik Meijering, Bjoern H. Menze, Karel G. M. Moons, Henning Müller, Brennan Nichyporuk, Felix Nickel, Jens Petersen, Susanne M. Rafelski, Nasir M. Rajpoot, Mauricio Reyes 0001, Michael A. Riegler, Nicola Rieke, Julio Saez-Rodriguez, Clara I. Sánchez, Shravya Shetty, Maarten van Smeden, Ronald M. Summers, Abdel A. Taha, Aleksei Tiulpin, Sotirios A. Tsaftaris, Ben Van Calster, Gaël Varoquaux, Manuel Wiesenfarth, Ziv R. Yaniv, Paul F. Jäger, and Lena Maier-Hein

Published
2023
Full Text
View/download PDF

21. MOOD 2020: A Public Benchmark for Out-of-Distribution Detection and Localization on Medical Images.

Author

David Zimmerer, Peter M. Full, Fabian Isensee, Paul Jäger 0002, Tim Adler, Jens Petersen, Gregor Köhler, Tobias Roß, Annika Reinke, Antanas Kascenas, Bjørn Sand Jensen, Alison Q. O'Neil, Jeremy Tan, Benjamin Hou, James Batten, Huaqi Qiu, Bernhard Kainz, Nina Shvetsova, Irina Fedulova, Dmitry V. Dylov, Baolun Yu, Jianyang Zhai, Jingtao Hu, Runxuan Si, Sihang Zhou 0001, Siqi Wang 0001, Xinyang Li, Xuerun Chen, Yang Zhao 0003, Sergio Naval Marimont, Giacomo Tarroni, Victor Saase, Lena Maier-Hein, and Klaus H. Maier-Hein

Published
2022
Full Text
View/download PDF

22. Abstract: MOOD 2020 - A Public Benchmark for Out-of-distribution Detection and Localization on Medical Images.

Author

David Zimmerer, Peter M. Full, Fabian Isensee, Paul Jäger 0002, Tim Adler, Jens Petersen, Gregor Köhler, Tobias Roß, Annika Reinke, Antanas Kascenas, Bjørn Sand Jensen, Alison Q. O'Neil, Jeremy Tan, Benjamin Hou, James Batten, Huaqi Qiu, Bernhard Kainz, Nina Shvetsova, Irina Fedulova, Dmitry V. Dylov, Baolun Yu, Jianyang Zhai, Jingtao Hu, Runxuan Si, Sihang Zhou 0001, Siqi Wang 0001, Xinyang Li, Xuerun Chen, Yang Zhao 0003, Sergio Naval Marimont, Giacomo Tarroni, Victor Saase, Lena Maier-Hein, and Klaus H. Maier-Hein

Published
2023
Full Text
View/download PDF

23. Comparative validation of machine learning algorithms for surgical workflow and skill analysis with the HeiChole benchmark.

Author

Martin Wagner 0001, Beat P. Müller-Stich, Anna Kisilenko, Duc Tran, Patrick Heger, Lars Mündermann, David M. Lubotsky, Benjamin Müller, Tornike Davitashvili, Manuela Capek, Annika Reinke, Carissa Reid, Tong Yu 0009, Armine Vardazaryan, Chinedu Innocent Nwoye, Nicolas Padoy, Xinyang Liu, Eung-Joo Lee, Constantin Disch, Hans Meine, Tong Xia, Fucang Jia, Satoshi Kondo, Wolfgang Reiter, Yueming Jin, Yonghao Long 0001, Meirui Jiang, Qi Dou 0001, Pheng-Ann Heng, Isabell Twick, Kadir Kirtaç, Enes Hosgor, Jon Lindström Bolmgren, Michael Stenzel, Björn von Siemens, Long Zhao, Zhenxiao Ge, Haiming Sun, Di Xie, Mengqi Guo, Daochang Liu, Hannes Götz Kenngott, Felix Nickel, Moritz von Frankenberg, Franziska Mathis-Ullrich, Annette Kopp-Schneider, Lena Maier-Hein, Stefanie Speidel, and Sebastian Bodenstedt

Published
2023
Full Text
View/download PDF

24. Beyond rankings: Learning (more) from algorithm validation.

Author

Tobias Roß, Pierangela Bruno, Annika Reinke, Manuel Wiesenfarth, Lisa Koeppel, Peter M. Full, Bünyamin Pekdemir, Patrick Godau, Darya Trofimova, Fabian Isensee, Tim J. Adler, Thuy Nuong Tran, Sara Moccia, Francesco Calimeri, Beat P. Müller-Stich, Annette Kopp-Schneider, and Lena Maier-Hein

Published
2023
Full Text
View/download PDF

27. Metrics reloaded: Pitfalls and recommendations for image analysis validation.

Author

Lena Maier-Hein, Annika Reinke, Evangelia Christodoulou, Ben Glocker, Patrick Godau, Fabian Isensee, Jens Kleesiek, Michal Kozubek 0001, Mauricio Reyes 0001, Michael A. Riegler, Manuel Wiesenfarth, Michael Baumgartner 0001, Matthias Eisenmann, Doreen Heckmann-Nötzel, A. Emre Kavur, Tim Rädsch, Minu Dietlinde Tizabi, Laura Ación, Michela Antonelli, Tal Arbel, Spyridon Bakas, Peter Bankhead, Arriel Benis, M. Jorge Cardoso, Veronika Cheplygina, Beth A. Cimini, Gary S. Collins, Keyvan Farahani, Bram van Ginneken, Daniel A. Hashimoto, Michael M. Hoffman, Merel Huisman, Pierre Jannin, Charles E. Kahn, Alexandros Karargyris, Alan Karthikesalingam, Hannes Kenngott, Annette Kopp-Schneider, Anna Kreshuk, Tahsin M. Kurç, Bennett A. Landman, Geert Litjens 0001, Amin Madani, Klaus H. Maier-Hein, Anne L. Martel, Peter Mattson, Erik Meijering, Bjoern H. Menze, David Moher, Karel G. M. Moons, Henning Müller, Felix Nickel, Brennan Nichyporuk, Jens Petersen, Nasir M. Rajpoot, Nicola Rieke, Julio Saez-Rodriguez, Clarisa Sánchez Gutiérrez, Shravya Shetty, Maarten van Smeden, Carole H. Sudre, Ronald M. Summers, Abdel A. Taha, Sotirios A. Tsaftaris, Ben Van Calster, Gaël Varoquaux, and Paul F. Jäger

Published
2022
Full Text
View/download PDF

28. Biomedical image analysis competitions: The state of current participation practice.

Author

Matthias Eisenmann, Annika Reinke, Vivienn Weru, Minu Dietlinde Tizabi, Fabian Isensee, Tim J. Adler, Patrick Godau, Veronika Cheplygina, Michal Kozubek 0001, Sharib Ali, Anubha Gupta, Jan Kybic, J. Alison Noble, Carlos Ortiz-de-Solórzano, Samiksha Pachade, Caroline Petitjean, Daniel Sage, Donglai Wei 0001, Elizabeth Wilden, Deepak Alapatt, Vincent Andrearczyk, Ujjwal Baid, Spyridon Bakas, Niranjan Balu, Sophia Bano, Vivek Singh Bawa, Jorge Bernal, Sebastian Bodenstedt, Alessandro Casella, Jinwook Choi, Olivier Commowick, Marie Daum, Adrien Depeursinge, Reuben Dorent, Jan Egger, Hannah Eichhorn, Sandy Engelhardt, Melanie Ganz, Gabriel Girard, Lasse Hansen, Mattias P. Heinrich, Nicholas Heller, Alessa Hering, Arnaud Huaulmé, Hyunjeong Kim, Bennett A. Landman, Hongwei Bran Li, Jianning Li, Jun Ma 0016, Anne L. Martel, and et al.

Published
2022
Full Text
View/download PDF

29. How to Exploit Weaknesses in Biomedical Challenge Design and Organization.

Author

Annika Reinke, Matthias Eisenmann, Sinan Onogur, Marko Stankovic, Patrick Scholz 0001, Peter M. Full, Hrvoje Bogunovic, Bennett A. Landman, Oskar Maier, Bjoern H. Menze, Gregory C. Sharp, Korsuk Sirinukunwattana, Stefanie Speidel, Fons van der Sommen, Guoyan Zheng, Henning Müller, Michal Kozubek 0001, Tal Arbel, Andrew P. Bradley, Pierre Jannin, Annette Kopp-Schneider, and Lena Maier-Hein

Published
2018
Full Text
View/download PDF

31. Common Limitations of Image Processing Metrics: A Picture Story.

Author

Annika Reinke, Matthias Eisenmann, Minu Dietlinde Tizabi, Carole H. Sudre, Tim Rädsch, Michela Antonelli, Tal Arbel, Spyridon Bakas, M. Jorge Cardoso, Veronika Cheplygina, Keyvan Farahani, Ben Glocker, Doreen Heckmann-Nötzel, Fabian Isensee, Pierre Jannin, Charles E. Kahn, Jens Kleesiek, Tahsin M. Kurç, Michal Kozubek 0001, Bennett A. Landman, Geert Litjens 0001, Klaus H. Maier-Hein, Bjoern H. Menze, Henning Müller, Jens Petersen, Mauricio Reyes 0001, Nicola Rieke, Bram Stieltjes, Ronald M. Summers, Sotirios A. Tsaftaris, Bram van Ginneken, Annette Kopp-Schneider, Paul Jäger 0002, and Lena Maier-Hein

Published
2021

32. Comparative Validation of Machine Learning Algorithms for Surgical Workflow and Skill Analysis with the HeiChole Benchmark.

Author

Martin Wagner 0001, Beat Peter Müller-Stich, Anna Kisilenko, Duc Tran, Patrick Heger, Lars Mündermann, David M. Lubotsky, Benjamin Müller, Tornike Davitashvili, Manuela Capek, Annika Reinke, Tong Yu 0009, Armine Vardazaryan, Chinedu Innocent Nwoye, Nicolas Padoy, Xinyang Liu, Eung-Joo Lee, Constantin Disch, Hans Meine, Tong Xia, Fucang Jia, Satoshi Kondo, Wolfgang Reiter, Yueming Jin, Yonghao Long 0001, Meirui Jiang, Qi Dou 0001, Pheng-Ann Heng, Isabell Twick, Kadir Kirtaç, Enes Hosgor, Jon Lindström Bolmgren, Michael Stenzel, Björn von Siemens, Hannes Götz Kenngott, Felix Nickel, Moritz von Frankenberg, Franziska Mathis-Ullrich, Lena Maier-Hein, Stefanie Speidel, and Sebastian Bodenstedt

Published
2021

33. The Medical Segmentation Decathlon.

Author

Michela Antonelli, Annika Reinke, Spyridon Bakas, Keyvan Farahani, Annette Kopp-Schneider, Bennett A. Landman, Geert Litjens 0001, Bjoern H. Menze, Olaf Ronneberger, Ronald M. Summers, Bram van Ginneken, Michel Bilello, Patrick Bilic, Patrick Ferdinand Christ, Richard K. G. Do, Marc Gollub, Stephan Heckers, Henkjan J. Huisman, William R. Jarnagin, Maureen McHugo, Sandy Napel, Jennifer Goli-Pernicka, Kawal S. Rhode, Catalina Tobon-Gomez, Eugene Vorontsov, James A. Meakin, Sébastien Ourselin, Manuel Wiesenfarth, Pablo Arbeláez, Byeonguk Bae, Sihong Chen, Laura Alexandra Daza, Jianjiang Feng, Baochun He, Fabian Isensee, Yuanfeng Ji, Fucang Jia, Namkug Kim, Ildoo Kim, Dorit Merhof, Akshay Pai, BeomHee Park, Mathias Perslev, Ramin Rezaiifar, Oliver Rippel, Ignacio Sarasua, Wei Shen, Jaemin Son, Christian Wachinger, Liansheng Wang, Yan Wang 0015, Yingda Xia, Daguang Xu, Zhanwei Xu, Yefeng Zheng 0001, Amber L. Simpson, Lena Maier-Hein, and M. Jorge Cardoso

Published
2021

35. The Federated Tumor Segmentation (FeTS) Challenge.

Author

Sarthak Pati, Ujjwal Baid, Maximilian Zenk, Brandon Edwards, Micah J. Sheller, G. Anthony Reina, Patrick Foley, Alexey Gruzdev, Jason Martin, Shadi Albarqouni, Yong Chen 0016, Russell Taki Shinohara, Annika Reinke, David Zimmerer, John B. Freymann, Justin S. Kirby, Christos Davatzikos, Rivka R. Colen, Aikaterini Kotrotsou, Daniel S. Marcus, Mikhail Milchenko, Arash Nazeri, Hassan M. Fathallah-Shaykh, Roland Wiest, András Jakab, Marc-André Weber, Abhishek Mahajan, Lena Maier-Hein, Jens Kleesiek, Bjoern H. Menze, Klaus H. Maier-Hein, and Spyridon Bakas

Published
2021

36. Heidelberg Colorectal Data Set for Surgical Data Science in the Sensor Operating Room.

Author

Lena Maier-Hein, Martin Wagner 0001, Tobias Roß, Annika Reinke, Sebastian Bodenstedt, Peter M. Full, Hellena Hempe, Diana Mîndroc-Filimon, Patrick Scholz 0001, Thuy Nuong Tran, Pierangela Bruno, Anna Kisilenko, Benjamin Müller, Tornike Davitashvili, Manuela Capek, Minu Tizabi, Matthias Eisenmann, Tim J. Adler, Janek Gröhl, Melanie Schellenberg, Silvia Seidlitz, T. Y. Emmy Lai, Veith Roethlingshoefer, Fabian Both, Sebastian Bittel, Marc Mengler, Martin Apitz, Stefanie Speidel, Hannes Götz Kenngott, and Beat P. Müller-Stich

Published
2020

37. Robust Medical Instrument Segmentation Challenge 2019.

Author

Tobias Roß, Annika Reinke, Peter M. Full, Martin Wagner 0001, Hannes Kenngott, Martin Apitz, Hellena Hempe, Diana Mîndroc-Filimon, Patrick Scholz 0001, Thuy Nuong Tran, Pierangela Bruno, Pablo Arbeláez, Gui-Bin Bian, Sebastian Bodenstedt, Jon Lindström Bolmgren, Laura Bravo Sánchez, Hua-Bin Chen, Cristina González, Dong Guo, Pål Halvorsen, Pheng-Ann Heng, Enes Hosgor, Zeng-Guang Hou, Fabian Isensee, Debesh Jha, Tingting Jiang 0001, Yueming Jin, Kadir Kirtaç, Sabrina Kletz, Stefan Leger, Zhixuan Li, Klaus H. Maier-Hein, Zhen-Liang Ni, Michael A. Riegler, Klaus Schoeffmann, Ruohua Shi, Stefanie Speidel, Michael Stenzel, Isabell Twick, Guotai Wang, Jiacheng Wang 0002, Liansheng Wang, Lu Wang, Yujie Zhang, Yan-Jie Zhou, Lei Zhu 0003, Manuel Wiesenfarth, Annette Kopp-Schneider, Beat Peter Müller-Stich, and Lena Maier-Hein

Published
2020

38. BIAS: Transparent reporting of biomedical image analysis challenges.

Author

Lena Maier-Hein, Annika Reinke, Michal Kozubek 0001, Anne L. Martel, Tal Arbel, Matthias Eisenmann, Allan Hanbury, Pierre Jannin, Henning Müller, Sinan Onogur, Julio Saez-Rodriguez, Bram van Ginneken, Annette Kopp-Schneider, and Bennett A. Landman

Published
2019

40. Comparative validation of multi-instance instrument segmentation in endoscopy: Results of the ROBUST-MIS 2019 challenge.

Author

Tobias Roß, Annika Reinke, Peter M. Full, Martin Wagner 0001, Hannes Kenngott, Martin Apitz, Hellena Hempe, Diana Mîndroc-Filimon, Patrick Scholz 0001, Thuy Nuong Tran, Pierangela Bruno, Pablo Arbeláez, Gui-Bin Bian, Sebastian Bodenstedt, Jon Lindström Bolmgren, Laura Bravo Sánchez, Hua-Bin Chen, Cristina González, Dong Guo, Pål Halvorsen, Pheng-Ann Heng, Enes Hosgor, Zeng-Guang Hou, Fabian Isensee, Debesh Jha, Tingting Jiang 0001, Yueming Jin, Kadir Kirtaç, Sabrina Kletz, Stefan Leger, Zhixuan Li, Klaus H. Maier-Hein, Zhen-Liang Ni, Michael A. Riegler, Klaus Schoeffmann, Ruohua Shi, Stefanie Speidel, Michael Stenzel, Isabell Twick, Guotai Wang, Jiacheng Wang 0002, Liansheng Wang, Lu Wang, Yujie Zhang, Yan-Jie Zhou, Lei Zhu 0003, Manuel Wiesenfarth, Annette Kopp-Schneider, Beat P. Müller-Stich, and Lena Maier-Hein

Published
2021
Full Text
View/download PDF

41. Comparative validation of machine learning algorithms for surgical workflow and skill analysis with the HeiChole benchmark

Author

Martin Wagner, Beat-Peter Müller-Stich, Anna Kisilenko, Duc Tran, Patrick Heger, Lars Mündermann, David M Lubotsky, Benjamin Müller, Tornike Davitashvili, Manuela Capek, Annika Reinke, Carissa Reid, Tong Yu, Armine Vardazaryan, Chinedu Innocent Nwoye, Nicolas Padoy, Xinyang Liu, Eung-Joo Lee, Constantin Disch, Hans Meine, Tong Xia, Fucang Jia, Satoshi Kondo, Wolfgang Reiter, Yueming Jin, Yonghao Long, Meirui Jiang, Qi Dou, Pheng Ann Heng, Isabell Twick, Kadir Kirtac, Enes Hosgor, Jon Lindström Bolmgren, Michael Stenzel, Björn von Siemens, Long Zhao, Zhenxiao Ge, Haiming Sun, Di Xie, Mengqi Guo, Daochang Liu, Hannes G. Kenngott, Felix Nickel, Moritz von Frankenberg, Franziska Mathis-Ullrich, Annette Kopp-Schneider, Lena Maier-Hein, Stefanie Speidel, Sebastian Bodenstedt, Laboratoire des sciences de l'ingénieur, de l'informatique et de l'imagerie (ICube), École Nationale du Génie de l'Eau et de l'Environnement de Strasbourg (ENGEES)-Université de Strasbourg (UNISTRA)-Institut National des Sciences Appliquées - Strasbourg (INSA Strasbourg), Institut National des Sciences Appliquées (INSA)-Institut National des Sciences Appliquées (INSA)-Institut National de Recherche en Informatique et en Automatique (Inria)-Les Hôpitaux Universitaires de Strasbourg (HUS)-Centre National de la Recherche Scientifique (CNRS)-Matériaux et Nanosciences Grand-Est (MNGE), Université de Strasbourg (UNISTRA)-Université de Haute-Alsace (UHA) Mulhouse - Colmar (Université de Haute-Alsace (UHA))-Institut National de la Santé et de la Recherche Médicale (INSERM)-Institut de Chimie du CNRS (INC)-Centre National de la Recherche Scientifique (CNRS)-Université de Strasbourg (UNISTRA)-Université de Haute-Alsace (UHA) Mulhouse - Colmar (Université de Haute-Alsace (UHA))-Institut National de la Santé et de la Recherche Médicale (INSERM)-Institut de Chimie du CNRS (INC)-Centre National de la Recherche Scientifique (CNRS)-Réseau nanophotonique et optique, and Université de Strasbourg (UNISTRA)-Université de Haute-Alsace (UHA) Mulhouse - Colmar (Université de Haute-Alsace (UHA))-Centre National de la Recherche Scientifique (CNRS)-Université de Strasbourg (UNISTRA)-Centre National de la Recherche Scientifique (CNRS)

Subjects
Surgical workflow analysis, Laparoscopic cholecystectomy, Radiological and Ultrasound Technology, Surgical data science, Informatique [cs]/Imagerie médicale, Endoscopic vision, DATA processing & computer science, [INFO.INFO-IM]Computer Science [cs]/Medical Imaging, Health Informatics, Radiology, Nuclear Medicine and imaging, Computer Vision and Pattern Recognition, ddc:004, Computer Graphics and Computer-Aided Design
Abstract

Purpose: Surgical workflow and skill analysis are key technologies for the next generation of cognitive surgical assistance systems. These systems could increase the safety of the operation through context-sensitive warnings and semi-autonomous robotic assistance or improve training of surgeons via data-driven feedback. In surgical workflow analysis up to 91% average precision has been reported for phase recognition on an open data single-center video dataset. In this work we investigated the generalizability of phase recognition algorithms in a multicenter setting including more difficult recognition tasks such as surgical action and surgical skill. Methods: To achieve this goal, a dataset with 33 laparoscopic cholecystectomy videos from three surgical centers with a total operation time of 22 h was created. Labels included framewise annotation of seven surgical phases with 250 phase transitions, 5514 occurences of four surgical actions, 6980 occurences of 21 surgical instruments from seven instrument categories and 495 skill classifications in five skill dimensions. The dataset was used in the 2019 international Endoscopic Vision challenge, sub-challenge for surgical workflow and skill analysis. Here, 12 research teams trained and submitted their machine learning algorithms for recognition of phase, action, instrument and/or skill assessment. Results: F1-scores were achieved for phase recognition between 23.9% and 67.7% (n = 9 teams), for instrument presence detection between 38.5% and 63.8% (n = 8 teams), but for action recognition only between 21.8% and 23.3% (n = 5 teams). The average absolute error for skill assessment was 0.78 (n = 1 team). Conclusion: Surgical workflow and skill analysis are promising technologies to support the surgical team, but there is still room for improvement, as shown by our comparison of machine learning algorithms. This novel HeiChole benchmark can be used for comparable evaluation and validation of future work. In future studies, it is of utmost importance to create more open, high-quality datasets in order to allow the development of artificial intelligence and cognitive robotics in surgery.

Published
2023
Full Text
View/download PDF

43. Is the winner really the best? A critical analysis of common research practice in biomedical image analysis competitions.

Author

Lena Maier-Hein, Matthias Eisenmann, Annika Reinke, Sinan Onogur, Marko Stankovic, Patrick Scholz 0001, Tal Arbel, Hrvoje Bogunovic, Andrew P. Bradley, Aaron Carass, Carolin Feldmann, Alejandro F. Frangi, Peter M. Full, Bram van Ginneken, Allan Hanbury, Katrin Honauer, Michal Kozubek 0001, Bennett A. Landman, Keno März, Oskar Maier, Klaus H. Maier-Hein, Bjoern H. Menze, Henning Müller, Peter F. Neher, Wiro J. Niessen, Nasir M. Rajpoot, Gregory C. Sharp, Korsuk Sirinukunwattana, Stefanie Speidel, Christian Stock, Danail Stoyanov, Abdel Aziz Taha, Fons van der Sommen, Ching-Wei Wang, Marc-André Weber, Guoyan Zheng, Pierre Jannin, and Annette Kopp-Schneider

Published
2018

45. Methods and open-source toolkit for analyzing and visualizing challenge results

Author

Matthias Eisenmann, Annika Reinke, Manuel Wiesenfarth, Lena Maier-Hein, M. Jorge Cardoso, Annette Kopp-Schneider, Laura Aguilera Saiz, and Bennett A. Landman

Subjects
0301 basic medicine, Creative visualization, Multidisciplinary, Computer science, Event (computing), media_common.quotation_subject, Science, Context (language use), Benchmarking, Data science, Article, 030218 nuclear medicine & medical imaging, Visualization, 03 medical and health sciences, 030104 developmental biology, 0302 clinical medicine, Image processing, Machine learning, Medicine, Medical imaging, Grand Challenges, media_common
Abstract

Grand challenges have become the de facto standard for benchmarking image analysis algorithms. While the number of these international competitions is steadily increasing, surprisingly little effort has been invested in ensuring high quality design, execution and reporting for these international competitions. Specifically, results analysis and visualization in the event of uncertainties have been given almost no attention in the literature. Given these shortcomings, the contribution of this paper is two-fold: (1) we present a set of methods to comprehensively analyze and visualize the results of single-task and multi-task challenges and apply them to a number of simulated and real-life challenges to demonstrate their specific strengths and weaknesses; (2) we release the open-source framework challengeR as part of this work to enable fast and wide adoption of the methodology proposed in this paper. Our approach offers an intuitive way to gain important insights into the relative and absolute performance of algorithms, which cannot be revealed by commonly applied visualization techniques. This is demonstrated by the experiments performed in the specific context of biomedical image analysis challenges. Our framework could thus become an important tool for analyzing and visualizing challenge results in the field of biomedical image analysis and beyond.

Published
2021

46. Common Pitfalls and Recommendations for Grand Challenges in Medical Artificial Intelligence

Author

Matthias Eisenmann, Lena Maier-Hein, Annika Reinke, and Minu D. Tizabi

Subjects
Biomedical Research, Point (typography), business.industry, Urology, Interpretation (philosophy), media_common.quotation_subject, 030232 urology & nephrology, Medical research, Domain (software engineering), 03 medical and health sciences, 0302 clinical medicine, Artificial Intelligence, 030220 oncology & carcinogenesis, Transparency (graphic), Medicine, Humans, Quality (business), Artificial intelligence, business, Algorithms, Grand Challenges, Interpretability, media_common
Abstract

With the impact of artificial intelligence (AI) algorithms on medical research on the rise, the importance of competitions for comparative validation of algorithms, so-called challenges, has been steadily increasing, to a point at which challenges can be considered major drivers of research, particularly in the biomedical image analysis domain. Given their importance, high quality, transparency, and interpretability of challenges is essential for good scientific practice and meaningful validation of AI algorithms, for instance towards clinical translation. This mini-review presents several issues related to the design, execution, and interpretation of challenges in the biomedical domain and provides best-practice recommendations. PATIENT SUMMARY: This paper presents recommendations on how to reliably compare the usefulness of new artificial intelligence methods for analysis of medical images.

Published
2021

47. Author Correction: Methods and open-source toolkit for analyzing and visualizing challenge results

Author

Annika Reinke, Laura Aguilera Saiz, Manuel Wiesenfarth, Annette Kopp-Schneider, Matthias Eisenmann, M. Jorge Cardoso, Bennett A. Landman, and Lena Maier-Hein

Subjects
Multidisciplinary, Correction method, Open source, Information retrieval, Computer science, Science, Medicine, Author Correction
Abstract

Grand challenges have become the de facto standard for benchmarking image analysis algorithms. While the number of these international competitions is steadily increasing, surprisingly little effort has been invested in ensuring high quality design, execution and reporting for these international competitions. Specifically, results analysis and visualization in the event of uncertainties have been given almost no attention in the literature. Given these shortcomings, the contribution of this paper is two-fold: (1) we present a set of methods to comprehensively analyze and visualize the results of single-task and multi-task challenges and apply them to a number of simulated and real-life challenges to demonstrate their specific strengths and weaknesses; (2) we release the open-source framework challengeR as part of this work to enable fast and wide adoption of the methodology proposed in this paper. Our approach offers an intuitive way to gain important insights into the relative and absolute performance of algorithms, which cannot be revealed by commonly applied visualization techniques. This is demonstrated by the experiments performed in the specific context of biomedical image analysis challenges. Our framework could thus become an important tool for analyzing and visualizing challenge results in the field of biomedical image analysis and beyond.

Published
2021

48. Medical Image Computing and Computer Assisted Intervention – MICCAI 2023 Workshops : ISIC 2023, Care-AI 2023, MedAGI 2023, DeCaF 2023, Held in Conjunction with MICCAI 2023, Vancouver, BC, Canada, October 8–12, 2023, Proceedings

Author

M. Emre Celebi, Md Sirajus Salekin, Hyunwoo Kim, Shadi Albarqouni, Catarina Barata, Allan Halpern, Philipp Tschandl, Marc Combalia, Yuan Liu, Ghada Zamzmi, Joshua Levy, Huzefa Rangwala, Annika Reinke, Diya Wynn, Bennett Landman, Won-Ki Jeong, Yiqing Shen, Zhongying Deng, Spyridon Bakas, Xiaoxiao Li, Chen Qin, Nicola Rieke, Holger Roth, Daguang Xu, M. Emre Celebi, Md Sirajus Salekin, Hyunwoo Kim, Shadi Albarqouni, Catarina Barata, Allan Halpern, Philipp Tschandl, Marc Combalia, Yuan Liu, Ghada Zamzmi, Joshua Levy, Huzefa Rangwala, Annika Reinke, Diya Wynn, Bennett Landman, Won-Ki Jeong, Yiqing Shen, Zhongying Deng, Spyridon Bakas, Xiaoxiao Li, Chen Qin, Nicola Rieke, Holger Roth, and Daguang Xu

Subjects
Image processing—Digital techniques, Computer vision
Abstract

This double volume set LNCS 14393-14394 constitutes the proceedings from the workshops held at the 26th International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI 2023 Workshops, which took place in Vancouver, BC, Canada, in October 2023. The 54 full papers together with 14 short papers presented in this volume were carefully reviewed and selected from 123 submissions from all workshops. The papers of the workshops are presenting the topical sections: Eighth International Skin Imaging Collaboration Workshop (ISIC 2023) First Clinically-Oriented and Responsible AI for Medical Data Analysis (Care-AI 2023) Workshop First International Workshop on Foundation Models for Medical Artificial General Intelligence (MedAGI 2023) Fourth Workshop on Distributed, Collaborative and Federated Learning (DeCaF 2023) First MICCAI Workshop on Time-Series Data Analytics and Learning First MICCAI Workshop on Lesion Evaluation and Assessment with Follow-Up (LEAF) AI For Treatment Response Assessment and predicTion Workshop (AI4Treat 2023) Fourth International Workshop on Multiscale Multimodal Medical Imaging (MMMI 2023) Second International Workshop on Resource-Effcient Medical Multimodal Medical Imaging Image Analysis (REMIA 2023)

Published
2023

49. Comparative validation of multi-instance instrument segmentation in endoscopy: Results of the ROBUST-MIS 2019 challenge

Author

Pablo Arbeláez, Annika Reinke, Sabrina Kletz, Yueming Jin, Zhen-Liang Ni, Fabian Isensee, Cristina González, Sebastian Bodenstedt, Pål Halvorsen, Lena Maier-Hein, Ruohua Shi, Beat P. Müller-Stich, Hannes Kenngott, Martin Wagner, Yan-Jie Zhou, Kadir Kirtac, Manuel Wiesenfarth, Stefanie Speidel, Stefan Leger, Zhixuan Li, Thuy Nuong Tran, Tingting Jiang, Peter M. Full, Klaus H. Maier-Hein, Patrick Scholz, Laura Bravo-Sánchez, Hua-Bin Chen, Yujie Zhang, Lei Zhu, Annette Kopp-Schneider, Zeng-Guang Hou, Diana Mindroc-Filimon, Liansheng Wang, Gutai Wang, Enes Hosgor, Hellena Hempe, Tobias Roß, Jon Lindström Bolmgren, Pierangela Bruno, Martin Apitz, Michael Riegler, Gui-Bin Bian, Lu Wang, Pheng-Ann Heng, Michael Stenzel, Klaus Schoeffmann, Debesh Jha, Dong Guo, Jiacheng Wang, and Isabell Twick

Subjects
Computer science, ComputingMethodologies_IMAGEPROCESSINGANDCOMPUTERVISION, Health Informatics, Machine learning, computer.software_genre, 030218 nuclear medicine & medical imaging, 03 medical and health sciences, 0302 clinical medicine, Market segmentation, Motion artifacts, Robustness (computer science), Image Processing, Computer-Assisted, Radiology, Nuclear Medicine and imaging, Segmentation, Radiological and Ultrasound Technology, business.industry, Medical instruments, Benchmarking, Computer Graphics and Computer-Aided Design, Video image, Laparoscopy, Computer Vision and Pattern Recognition, Artificial intelligence, Artifacts, business, computer, Algorithms, 030217 neurology & neurosurgery, Test data
Abstract

Intraoperative tracking of laparoscopic instruments is often a prerequisite for computer and robotic-assisted interventions. While numerous methods for detecting, segmenting and tracking of medical instruments based on endoscopic video images have been proposed in the literature, key limitations remain to be addressed: Firstly, robustness, that is, the reliable performance of state-of-the-art methods when run on challenging images (e.g. in the presence of blood, smoke or motion artifacts). Secondly, generalization; algorithms trained for a specific intervention in a specific hospital should generalize to other interventions or institutions. In an effort to promote solutions for these limitations, we organized the Robust Medical Instrument Segmentation (ROBUST-MIS) challenge as an international benchmarking competition with a specific focus on the robustness and generalization capabilities of algorithms. For the first time in the field of endoscopic image processing, our challenge included a task on binary segmentation and also addressed multi-instance detection and segmentation. The challenge was based on a surgical data set comprising 10,040 annotated images acquired from a total of 30 surgical procedures from three different types of surgery. The validation of the competing methods for the three tasks (binary segmentation, multi-instance detection and multi-instance segmentation) was performed in three different stages with an increasing domain gap between the training and the test data. The results confirm the initial hypothesis, namely that algorithm performance degrades with an increasing domain gap. While the average detection and segmentation quality of the best-performing algorithms is high, future research should concentrate on detection and segmentation of small, crossing, moving and transparent instrument(s) (parts).

Published
2020

50. BIAS: Transparent reporting of biomedical image analysis challenges

Author

Annika Reinke, Bram van Ginneken, Annette Kopp-Schneider, Tal Arbel, Anne L. Martel, Matthias Eisenmann, Henning Müller, Allan Hanbury, Pierre Jannin, Sinan Onogur, Julio Saez-Rodriguez, Bennett A. Landman, Michal Kozubek, Lena Maier-Hein, German Cancer Research Center - Deutsches Krebsforschungszentrum [Heidelberg] (DKFZ), Masaryk University [Brno] (MUNI), University of Toronto, McGill University = Université McGill [Montréal, Canada], Vienna University of Technology (TU Wien), Laboratoire Traitement du Signal et de l'Image (LTSI), Université de Rennes 1 (UR1), Université de Rennes (UNIV-RENNES)-Université de Rennes (UNIV-RENNES)-Institut National de la Santé et de la Recherche Médicale (INSERM), University of Applied Sciences and Arts of Western Switzerland (HES-SO), University of Geneva [Switzerland], Heidelberg University, Radboud University Medical Center [Nijmegen], Vanderbilt University [Nashville], ERC-2015-StG-37960, European Research Council, RGPIN-2016-06283, Natural Sciences and Engineering Research Council of Canada, 705772, Canadian Cancer Society, CZ.02.1.01/0.0/0.0/16_013/0001775, Ministerstvo Školství, Mládeže a Tělovýchovy, 1452485, National Science Foundation, R01-EB017230-01A1, NIH Clinical Center, Université de Rennes (UR)-Institut National de la Santé et de la Recherche Médicale (INSERM), Université de Genève = University of Geneva (UNIGE), and HAL UR1, Admin

Subjects
FOS: Computer and information sciences, Biomedical Research, Computer science, Computer Vision and Pattern Recognition (cs.CV), media_common.quotation_subject, Computer Science - Computer Vision and Pattern Recognition, Good scientific practice, Biomedical challenges, Health Informatics, Guideline, Biomedical image analysis, Article, Field (computer science), Task (project management), Computer Science - Computers and Society, 03 medical and health sciences, 0302 clinical medicine, All institutes and research themes of the Radboud University Medical Center, Computers and Society (cs.CY), FOS: Electrical engineering, electronic engineering, information engineering, Humans, Radiology, Nuclear Medicine and imaging, Quality (business), Set (psychology), 030304 developmental biology, Interpretability, media_common, [SDV.IB] Life Sciences [q-bio]/Bioengineering, 0303 health sciences, Radiological and Ultrasound Technology, Image and Video Processing (eess.IV), Reproducibility of Results, Benchmarking, Electrical Engineering and Systems Science - Image and Video Processing, Prognosis, Computer Graphics and Computer-Aided Design, Data science, Checklist, 3. Good health, Transparency (graphic), [SDV.IB]Life Sciences [q-bio]/Bioengineering, Computer Vision and Pattern Recognition, 030217 neurology & neurosurgery, Rare cancers Radboud Institute for Health Sciences [Radboudumc 9]
Abstract

Highlights • First guideline for reporting biomedical image analysis challenges • Includes checklist for authors and reviewers, Graphical abstract, The number of biomedical image analysis challenges organized per year is steadily increasing. These international competitions have the purpose of benchmarking algorithms on common data sets, typically to identify the best method for a given problem. Recent research, however, revealed that common practice related to challenge reporting does not allow for adequate interpretation and reproducibility of results. To address the discrepancy between the impact of challenges and the quality (control), the Biomedical Image Analysis ChallengeS (BIAS) initiative developed a set of recommendations for the reporting of challenges. The BIAS statement aims to improve the transparency of the reporting of a biomedical image analysis challenge regardless of field of application, image modality or task category assessed. This article describes how the BIAS statement was developed and presents a checklist which authors of biomedical image analysis challenges are encouraged to include in their submission when giving a paper on a challenge into review. The purpose of the checklist is to standardize and facilitate the review process and raise interpretability and reproducibility of challenge results by making relevant information explicit.

Published
2020

Catalog

Books, media, physical & digital resources
See catalog results