Search

Amazon Mechanical Turk (MTurk) is one of the most popular crowdsourcing platforms for online research. Although the recent MTurk data quality crisis has raised serious concerns regarding its utility, this issue can be mitigated by including quality control methods. To date, only a few studies have compared the effectiveness of the methods in preserving an experimental effect. To provide empirical guidance on which quality control methods to use for MTurk research, this study compared different methods on their effectiveness in preserving the magnitude effect in delay discounting, a robust finding in the behavioral economics literature. Two groups of MTurkers, whose HIT approval rate (%) was between 95 and 98 and 99 and 100, were recruited in October 2021. They completed a delay discounting task evaluating the magnitude effect and questions from different quality control methods. The results showed that the group with higher HIT approval rates (i.e., 99%--100%) showed a greater magnitude effect and were more likely to pass the quality control methods than those with lower rates. In addition, the majority of the quality control methods were able to preserve the magnitude effect at the expense of sample sizes. The best combinations of quality control methods that optimally maintained the effect were explored with a discussion of their implications. Although improving data quality is imperative, caution should be exercised as the quality control methods may potentially bias the sample in an undesirable manner (e.g., excluding individuals who show steep delay discounting). Keywords MTurk * Data quality * Online research * Delay discounting * Magnitude effect, Introduction The use of online crowdsourcing platforms to collect behavioral data has become popular in social science research in recent decades. These platforms provide evident scientific benefits, including efficiency and [...]

Previous discounting research has focused on relatively simple situations (e.g., choosing between immediate and delayed gains, or between immediate and delayed losses) and the relations among amount, delay, and subjective value in such situations are now well established. Many everyday choice situations, however, are more complex, involving alternatives that combine gains and losses. In two experiments, participants discounted a monetary outcome that combined an immediate gain with a delayed loss. Consistent with the discounting framework, the value of the combination was approximately equal to the difference between the (undiscounted) value of the immediate gain and the (discounted) value of the delayed loss. More precise predictions of the relative subjective value of the delayed loss component of a combination were given by the same hyperboloid function that describes discounting in simple choice situations where outcomes only involve gains or losses, not both. As in previous studies, when the outcome was a simple one consisting of only a delayed loss, discounting was not affected by the amount of that loss. Surprisingly, however, when the loss was preceded by a gain, the degree to which the combination was discounted was affected by the amount of the loss. Notably, discounting of the delayed loss component of the combination decreased with the amount of the loss in a fashion similar to that observed with simple delayed gains, where smaller amounts are discounted more steeply than larger amounts. Choice situations involving immediate gains followed by delayed losses pose iconic self‐control problems, and the present findings support the application of the discounting framework to these important everyday problems. [ABSTRACT FROM AUTHOR]