1. Active label cleaning for improved dataset quality under resource constraints
- Author
-
Bernhardt, Melanie, Castro, Daniel C., Tanno, Ryutaro, Schwaighofer, Anton, Tezcan, Kerem C., Monteiro, Miguel, Bannur, Shruthi, Lungren, Matthew, Nori, Aditya, Glocker, Ben, Alvarez-Valle, Javier, and Oktay, Ozan
- Subjects
Computer Science - Computer Vision and Pattern Recognition ,Computer Science - Machine Learning - Abstract
Imperfections in data annotation, known as label noise, are detrimental to the training of machine learning models and have an often-overlooked confounding effect on the assessment of model performance. Nevertheless, employing experts to remove label noise by fully re-annotating large datasets is infeasible in resource-constrained settings, such as healthcare. This work advocates for a data-driven approach to prioritising samples for re-annotation - which we term "active label cleaning". We propose to rank instances according to estimated label correctness and labelling difficulty of each sample, and introduce a simulation framework to evaluate relabelling efficacy. Our experiments on natural images and on a new medical imaging benchmark show that cleaning noisy labels mitigates their negative impact on model training, evaluation, and selection. Crucially, the proposed active label cleaning enables correcting labels up to 4 times more effectively than typical random selection in realistic conditions, making better use of experts' valuable time for improving dataset quality., Comment: Accepted for publication in Nature Communications
- Published
- 2021
- Full Text
- View/download PDF