Search

Crowdsourcing plays a vital role in today's AI industry. However, existing crowdsourcing research mainly focuses on those simple tasks that are often formulated as label classification, while complex open-ended tasks such as question answering and translation have not received much attention. Such tasks usually have open solution spaces and non-unique true answers, which pose great challenges for designing effective crowdsourcing algorithms. In this work, we are concerned specifically with complex text annotation crowdsourcing tasks, where each answer of a task is in the form of free text. We propose an error consistency-based approach to inferring a satisfying result from a set of open-ended answers. First, each answer is represented with two vectors that capture the local word collocation and the global sentence semantics respectively. Second, the true answer is approximated by the sum of the answer vectors weighted by the reciprocals of their respective errors. Third, an algorithm called AEC (A ggregation based on E rror C onsistency) is designed to infer the aggregated result by maximizing the consistency of the errors of an answer in two vector spaces. Experimental results on two datasets demonstrate the effectiveness of our approach. [ABSTRACT FROM AUTHOR]

In contrast to the traditional view that individuals should work independently to realize the crowd wisdom, crowdsourcing workers often collaborate with each other in task processing either explicitly or implicitly. Some may even collude for obtaining rewards easily, for example, by plagiarizing others' answers. Collusion behavior sabotages the independency among workers, and will subvert the benefits of task redundancy that is commonly adopted in crowdsourcing. Therefore, dealing with collusion is critical for ensuring the quality of crowdsourcing. Existing work usually treats all the collusive answers as being harmful, thus simply filters away them once they are detected. However, it is not always the best strategy in practice. In particular, when the collusive answers are plagiarized from a worker with good ability, utilizing them instead of simple elimination can benefit the result quality. In this work, we first propose a collusion-aware framework for detecting and handling collusion in crowdsourcing properly. Second, we design a collusion detection method based on the statistical test of the consistency of workers' answers across tasks. Third, we provide a theoretical means to determine when collusive answers should be kept and utilized, then we design a collusion-aware answer aggregation method. Finally, we conducted thorough evaluation with both synthetic and real-world datasets, and the results demonstrate the effectiveness of our approach. [ABSTRACT FROM AUTHOR]

Many result inference methods have been proposed to address the quality-control problem in crowdsourcing. However, existing methods are ineffective for context-sensitive tasks ( CSTs ), e.g., handwriting recognition, translation, speech transcription, where context correlation within a task cannot be ignored for two reasons. Firstly, it is ineffective to crowdsource a whole CST (e.g., recognizing handwritten texts) and use task-level inference methods to infer the answer, because it is rather hard to correctly complete a whole complicated task. Secondly, although a CST is composed of a set of atomic subtasks (e.g., recognizing a handwritten word), it is unsuitable to split it into multiple subtasks and adopt a subtask-level inference algorithm to infer the result, because this will lose the context correlation (e.g., phrases) among subtasks and increase the difficulty to complete a task. Thus it calls for a new approach to handling CSTs . In this work, we study the result inference problem for CSTs and propose a context-aware inference algorithm. We design an inference algorithm by incorporating the context information. Furthermore, we introduce an iterative method to improve the quality. The results of experiments on real-world CSTs demonstrated the superiority of our approach compared with the state-of-the-art methods. [ABSTRACT FROM AUTHOR]