Back to Search
Start Over
Statistical Decision Making for Optimal Budget Allocation in Crowd Labeling
- Source :
- SSRN Electronic Journal.
- Publication Year :
- 2014
- Publisher :
- Elsevier BV, 2014.
-
Abstract
- In crowd labeling, a large amount of unlabeled data instances are outsourced to a crowd of workers. Workers will be paid for each label they provide, but the labeling requester usually has only a limited amount of the budget. Since data instances have different levels of labeling difficulty and workers have different reliability, it is desirable to have an optimal policy to allocate the budget among all instance-worker pairs such that the overall labeling accuracy is maximized. We consider categorical labeling tasks and formulate the budget allocation problem as a Bayesian Markov decision process (MDP), which simultaneously conducts learning and decision making. Using the dynamic programming (DP) recurrence, one can obtain the optimal allocation policy. However, DP quickly becomes computationally intractable when the size of the problem increases. To solve this challenge, we propose a computationally efficient approximate policy, called optimistic knowledge gradient policy. Our MDP is a quite general framework, which applies to both pull crowdsourcing marketplaces with homogeneous workers and push marketplaces with heterogeneous workers. It can also incorporate the contextual information of instances when they are available. The experiments on both simulated and real data show that the proposed policy achieves a higher labeling accuracy than other existing policies at the same budget level.<br />Comment: 39 pages
- Subjects :
- FOS: Computer and information sciences
Mathematical optimization
Computer science
business.industry
Bayesian probability
Machine Learning (stat.ML)
Crowdsourcing
Machine Learning (cs.LG)
Dynamic programming
Computer Science - Learning
Statistics - Machine Learning
Optimization and Control (math.OC)
FOS: Mathematics
Contextual information
Markov decision process
business
Mathematics - Optimization and Control
Categorical variable
Budget allocation
Reliability (statistics)
Subjects
Details
- ISSN :
- 15565068
- Database :
- OpenAIRE
- Journal :
- SSRN Electronic Journal
- Accession number :
- edsair.doi.dedup.....2395979984d4524e1eb7b12cb5d9fd71