Back to Search
Start Over
Reputation as a sufficient condition for data quality on Amazon Mechanical Turk
- Source :
- Behavior Research Methods, 46(4), 1023-1031. Springer
- Publication Year :
- 2013
-
Abstract
- Data quality is one of the major concerns of using crowdsourcing websites such as Amazon Mechanical Turk (MTurk) to recruit participants for online behavioral studies. We compared two methods for ensuring data quality on MTurk: attention check questions (ACQs) and restricting participation to MTurk workers with high reputation (above 95% approval ratings). In Experiment 1, we found that high-reputation workers rarely failed ACQs and provided higher-quality data than did low-reputation workers; ACQs improved data quality only for low-reputation workers, and only in some cases. Experiment 2 corroborated these findings and also showed that more productive high-reputation workers produce the highest-quality data. We concluded that sampling high-reputation workers can ensure high-quality data without having to resort to using ACQs, which may lead to selection bias if participants who fail ACQs are excluded post-hoc.
- Subjects :
- FOS: Computer and information sciences
Research design
Artificial Intelligence and Image Processing
media_common.quotation_subject
Internet privacy
Online research
Experimental and Cognitive Psychology
Crowdsourcing
Arts and Humanities (miscellaneous)
Amazon Mechanical Turk
Developmental and Educational Psychology
data quality
Humans
General Psychology
BEHAVIORAL RESEARCH, DATA COLLECTION, HUMANS, INTERNET, RESEARCH DESIGN, CROWDSOURCING
media_common
Selection bias
Internet
Data collection
Amazon rainforest
business.industry
Data Collection
Patient Selection
reputation
Online research methods
Research Design
Data quality
Psychology (miscellaneous)
business
Psychology
Reputation
Behavioral Research
Subjects
Details
- ISSN :
- 15543528 and 1554351X
- Volume :
- 46
- Issue :
- 4
- Database :
- OpenAIRE
- Journal :
- Behavior research methods
- Accession number :
- edsair.doi.dedup.....0a4e6d24ef212b948848a9855cd80f23