Research Article: Common (Mis)Beliefs about Memory: A Replication and Comparison of Telephone and Mechanical Turk Survey Methods

Date Published: December 18, 2012

Publisher: Public Library of Science

Author(s): Daniel J. Simons, Christopher F. Chabris, Jan de Fockert.


Incorrect beliefs about memory have wide-ranging implications. We recently reported the results of a survey showing that a substantial proportion of the United States public held beliefs about memory that conflicted with those of memory experts. For that survey, respondents answered recorded questions using their telephone keypad. Although such robotic polling produces reliable results that accurately predicts the results of elections, it suffers from four major drawbacks: (1) telephone polling is costly, (2) typically, less than 10 percent of calls result in a completed survey, (3) calls do not reach households without a landline, and (4) calls oversample the elderly and undersample the young. Here we replicated our telephone survey using Amazon Mechanical Turk (MTurk) to explore the similarities and differences in the sampled demographics as well as the pattern of results. Overall, neither survey closely approximated the demographics of the United States population, but they differed in how they deviated from the 2010 census figures. After weighting the results of each survey to conform to census demographics, though, the two approaches produced remarkably similar results: In both surveys, people averaged over 50% agreement with statements that scientific consensus shows to be false. The results of this study replicate our finding of substantial discrepancies between popular beliefs and those of experts and shows that surveys conducted on MTurk can produce a representative sample of the United States population that generates results in line with more expensive survey techniques.

Partial Text

A substantial proportion of the United States public holds beliefs about memory that conflict with well-established expert consensus. In a nationally representative telephone survey with a nominal sample of 1500 people, on average 60.4% of respondents agreed with six statements that were almost universally rejected by a sample of memory experts [1]. This nationally representative sample produced results comparable to earlier telephone surveys of jurors and police as well as classroom and in-person surveys of students and the public [2], [3], [4], [5], [6], [7], showing that mistaken intuitions about memory are pervasive [8], [9]. For decisions that rely on intuitions about what people should remember and how well they should remember it, these mistaken beliefs can have devastating consequences. For example, according to the Innocence Project, many of those exonerated from death row following DNA testing were convicted on the basis of flawed eyewitness testimony, and jurors evaluate the credibility of that evidence based on their often mistaken understanding of the workings of memory [6].

Our earlier phone survey found high rates of agreement with statements that counter established scientific consensus, revealing widespread misconceptions about the workings of memory [1]. We found the same pattern with our MTurk sample, with roughly comparable levels of agreement for all our items. Before comparing the results of the two surveys, we address the central question of this study: What is the relative efficiency and accuracy of the sampling procedure achieved via telephone surveys and MTurk? To that end, we first compare the raw demographics of the samples in each survey.

The goals of this study were twofold: (1) to explore whether we could replicate the pattern of mistaken beliefs about memory found in our earlier telephone survey using a different sampling method, and (2) to compare the demographic characteristics of a sample of self-selected Mechanical Turk participants to those of a nationally representative telephone survey to determine whether MTurk could be used to obtain a nationally representative sample of the United States population.