Lookups and Measured Knowledge

How do survey respondents who look up the answers affect measures of knowledge? This paper derives the "lookup bias'' in survey measures of knowledge. Lookup bias has two key components: the prevalence of lookups and the "lookup yield," i.e. the average lookup's effect on measured knowledge. Even when lookups are common, the lookup yield is small when information is scarce or inaccurate, as well as when respondents use lookups to confirm what they already know rather than to fill in ignorance. To test these insights, I develop an instrumental variables method for estimating the lookup yield and apply it to the study of politics. On standard political knowledge questions, the yield is about 0.5, meaning that each lookup results in about 0.5 additional correct answers. Variation across questions in the lookup yield appears to be related to variation in the availability of accurate information and the probability of guessing correctly. For example, on "catch" questions that are designed to be easy to look up and hard to guess, the yield is close to 1. The results suggest that in most settings, lookup bias is considerably smaller than the proportion of respondents who look up the answer.