Date Published: February 21, 2012
Publisher: Public Library of Science
Author(s): Ofra Amir, David G. Rand, Ya’akov Kobi Gal, Matjaz Perc. http://doi.org/10.1371/journal.pone.0031461
Online labor markets such as Amazon Mechanical Turk (MTurk) offer an unprecedented opportunity to run economic game experiments quickly and inexpensively. Using Mturk, we recruited 756 subjects and examined their behavior in four canonical economic games, with two payoff conditions each: a stakes condition, in which subjects’ earnings were based on the outcome of the game (maximum earnings of $1); and a no-stakes condition, in which subjects’ earnings are unaffected by the outcome of the game. Our results demonstrate that economic game experiments run on MTurk are comparable to those run in laboratory settings, even when using very low stakes.
Online labor markets such as Amazon Mechanical Turk (MTurk) are internet marketplaces in which people can complete short tasks (typically 5 minutes or less) in exchange for small amounts of money (typically $1 or less). MTurk is becoming increasingly popular as a platform for conducting experiments across the social sciences –. In particular, MTurk offers an unprecedented opportunity to run incentivized economic game experiments quickly and inexpensively. Recent work has replicated classical findings such as framing and priming on MTurk –, found a high level of test-retest reliability on Mturk –, and shown quantitative agreement in behavior between MTurk and the physical laboratory , . Yet concerns remain regarding the low stakes typically used in MTurk experiments.
This research was approved by the committee on the use of human subjects in research of Harvard University, application number F17468-103. Informed consent was obtained from all subjects.
As can be seen in Figure 1, introducing stakes altered the distribution of offers in the DG, significantly reducing the average offer (no-stakes = 43.8%, stakes = 33.2%, ). In the UG, we found a marginally significant positive effect of stakes on Player 1 proposals (no-stakes = 46.1%, stakes = 49.7%, ). Given the small effect size and borderline significant p-value, we conclude that stakes have little effect on P1 offers in the UG. We also find no significant effect on Player 2 MAOs in the UG (excluding inconsistent players) (). However, we do find a significantly higher proportion of inconsistent Player 2′s in the no-stakes condition compared to the stakes condition ( test, ). As a result, we also find a significant effect of stakes on Player 2 rejection rates for some Player 1 offers in the UG when including inconsistent players ( for the 30% offer, and for all offers above 60%). There was no significant effect of stakes on transfers in the TG (), back-transfers in the TG ( for all possible Player 1 transfers), and contributions in the PGG (). We also test whether the variance in behavior differs between the stakes and no-stakes conditions using Levene’s F-test. Consistent with our results above, we find that the variance in DG donations is significantly smaller in the stakes condition compared to the no-stakes condition (), but that adding stakes did not have an effect on the variance of offers () and MAOs in the UG (), transfers () and back-transfers in the TG ( for back-transfers on all Player 1 transfers, except for the transfer of 25% where the variance of Player 2 back-transfers in the stakes condition was marginally higher, ), and contributions in the PGG ().