Noise filtering to improve data and model quality for crowdsourcing

Crowdsourcing services provide an easy means of acquiring labeled training data for supervised learning. However, the labels provided by a single crowd worker are often unreliable. Repeated labeling can be used to solve this problem. After multiple labels have been acquired by repeated labeling for each instance, in general consensus methods are used to obtain the integrated labels of instances. Although consensus methods are effective in practice, it cannot be denied that a level of noise still exists in the set of integrated labels. In this study, an attempt was made to employ noise filters to delete the noise in integrated labels, and consequently, enhance the training data and model quality. In fact, noise handling is a relatively mature field in the machine learning community, and many noise filters for deleting label noise have been presented in the past. However, to the best of our knowledge, in very few studies was noise filtering used to improve crowdsourcing learning. Therefore, in this study we empirically investigated the performance of noise filters in terms of improving crowdsourcing learning. Thus, in this paper some existing noise filters presented in previous papers are reviewed and their experimental application to crowdsourcing learning tasks is described. Experimental results based on 14 benchmark UCI data sets and three real-world data sets show that these noise filters can significantly reduce the noise level in integrated labels and thereby considerably enhance the performance of target classifiers.