1. Discordance, accuracy and reproducibility study of pathologists' diagnosis of melanoma and melanocytic tumors.
- Author
-
Haggenmüller S, Wies C, Abels J, Winterstein JT, Heinlein L, Nogueira Garcia C, Utikal JS, Wohlfeil SA, Meier F, Hobelsberger S, Gellrich FF, Sergon M, Hauschild A, French LE, Heinzerling L, Schlager JG, Ghoreschi K, Schlaak M, Hilke FJ, Poch G, Korsing S, Sarfert C, Berking C, Heppt MV, Erdmann M, Haferkamp S, Drexler K, Schadendorf D, Sondermann W, Goebeler M, Schilling B, Kather JN, Fröhling S, Llamas-Velasco M, Requena LC, Ferrara G, Fernandez-Figueras M, Fraitag S, Müller CSL, Starz H, Kutzner H, Barnhill R, Carr R, Resnik KS, Braun SA, Holland-Letz T, and Brinker TJ
- Subjects
- Humans, Reproducibility of Results, Female, Male, Middle Aged, Adult, Dermoscopy methods, Germany, Aged, Prospective Studies, Melanoma diagnosis, Melanoma pathology, Melanoma diagnostic imaging, Pathologists, Skin Neoplasms pathology, Skin Neoplasms diagnosis, Observer Variation
- Abstract
Accurate melanoma diagnosis is crucial for patient outcomes and reliability of AI diagnostic tools. We assess interrater variability among eight expert pathologists reviewing histopathological images and clinical metadata of 792 melanoma-suspicious lesions prospectively collected at eight German hospitals. Moreover, we provide access to the largest panel-validated dataset featuring dermoscopic and histopathological images with metadata. Complete agreement is achieved in 53.5% of cases (424/792), and a majority vote ( ≥ five pathologists) in 90.9% (720/792). Considerable discordance is observed for non-invasive melanomas (complete agreement in only 10/73 cases). The expert panel disagrees with the local pathologists' and dermatologists' diagnoses in 14.9% and 33.5% of cases, respectively. This variability highlights the diagnostic challenges of early-stage melanomas and the need to reconsider how ground truth is established in routine care and AI research. Including at least two pathologists or virtual panels may contribute to more consistent diagnostic results., Competing Interests: Competing interests: Prof Utikal reported personal fees from Amgen, Bristol Myers Squibb, GSK, Immunocore, LEO Pharma, Merck Sharp & Dohme, Novartis, Pierre Fabre, Rheacell, Roche, and Sanofi outside the submitted work. Dr Wohlfeil received honoraria from Bristol Myers Squibb, Novartis, and Sun Pharma outside the submitted work. Prof Meier reported grants from Novartis and Roche and others (travel support or/and speaker’s fees or/and advisor’s honoraria) from BMS, MSD, and Pierre Fabre outside the submitted work. Dr Hobelsberger reported clinical trial support and advisor’s honoraria from Almirall, speaker’s honoraria from Almirall, UCB, and AbbVie, and travel support from UCB, Janssen Cilag, Almirall, Novartis, Lilly, LEO Pharma, and AbbVie outside the submitted work. Prof Heinzerling reported other (clinical studies) from BMS, MSD, Pierre Fabre, Replimune, and Sanofi; personal fees from Biomedx, BMS, MSD, Sun, Pierre Fabre, Novartis, and Sanofi; and grants from Therakos outside the submitted work. Prof Haferkamp reported speaker’s fees and advisor’s honoraria from BMS, MSD, Novartis, and Pierre Fabre outside the submitted work. Dr Schlaak reported personal fees from BMS, Novartis, Immunocore, Sun Pharma, MSD, Recordati, and Sanofi Aventis outside the submitted work. Prof Berking reported personal fees from BMS, MSD, InflaRx, Novartis, Sanofi, LEO Pharma, Almirall Hermal, Pierre Fabre, Immunocore, and Delcath outside the submitted work. Dr Erdmann received travel support and speaker’s honoraria from Immunocore, Novartis, Pierre Fabre, and Sanofi outside the submitted work. Dr Sondermann reported grants from Almirall, Novartis, and Medi GmbH; and personal fees from AbbVie, Almirall, BMS, Boehringer Ingelheim, Celgene, Janssen, LEO Pharma, Lilly, Novartis, Pfizer, Sanofi Genzyme, and UCB outside the submitted work. Prof Goebeler reported grants for clinical studies from Argenx, Novartis, Janssen, and Galderma; personal fees from Almirall (consulting), Janssen (advisory board, speaker), GSK (advisory board, speaker), and Lilly (speaker) outside the submitted work. Dr Llamas-Velasco reported fees as an advisory board member, consultant, research support, participation in clinical trials, and honorary for speaking with the following pharmaceutical companies: Abbvie, Almirall, Amgen, Boehringer, BMS, Celgene, Janssen, Leo, Lilly, Kyowa kirin, Novartis and UCB, outside the submitted work. Dr. Kather declares consulting services for Owkin, France; DoMore Diagnostics, Norway; Panakeia, UK; AstraZeneca, UK; Scailyte, Switzerland; Mindpeak, Germany; and MultiplexDx, Slovakia. Furthermore, he holds shares in StratifAI GmbH, Germany, has received a research grant from GSK, and has received honoraria from AstraZeneca, Bayer, Eisai, Janssen, MSD, BMS, Roche, Pfizer, and Fresenius. Dr Gabriela Poch received travel support and speaker’s honoraria from MSD, BMS, Novartis, Sun Pharma, and Amgen outside the submitted work. Dr Brinker reported being the owner of Smart Health Heidelberg GmbH outside the submitted work. The remaining authors declare no competing interests., (© 2025. The Author(s).)
- Published
- 2025
- Full Text
- View/download PDF