Quantifying Worker Reliability for Crowdsensing Applications: Robust Feedback Rating and Convergence

Worker reliability estimation is fundamental for crowdsensing applications. This paper studies a robust feedback rating approach to estimate worker reliability. In this approach, the requester provides a feedback rating to reflect the quality of the sensor data submitted by each worker. The aggregation of each worker's historical feedback ratings serves as a reliability estimate. The challenges are: (1) Feedback ratings are subjected to noise or bias; (2) Workers cognitive bias in task selection leads to higher sensor data quality variations. We develop a mathematical model to quantify the degree of rating bias and degree of cognitive bias. We also derive sufficient conditions, under which the aggregate rating is asymptotically accurate in estimating worker reliability, via stochastic approximation techniques. These conditions identify a class of asymptotically accurate rating aggregation rules for crowdsensing applications. We further derive the minimum number of ratings needed to guarantee a given reliability estimation accuracy, via martingale theory. Via extensive experiments: (1) We reveal fundamental understandings on how various factor such as rating bias influence the minimum number of ratings needed to achieve certain accuracy; (2) Our feedback rating approach improves air quality index estimation accuracy by as high as 50% over the URP algorithm.