Do Attempts to Improve Respondent Attention Increase Social Desirability Bias?

In this research note, we investigate the effectiveness of warnings as a method for increasing respondent motivation and decreasing survey satisficing. Four different styles of warning messages are examined in data from a randomized survey experiment conducted on the Internet. The analysis shows that three of the four warnings significantly improve respondent engagement. There is some evidence, however, that warning messages increase socially desirable responding (SDR) for certain types of people. We conclude that warnings can be a useful method for increasing attention, but that researchers should first establish that these protocol do not have unintended consequences, either for the entire sample or for theoretically relevant subgroups. Social scientists increasingly collect data online through self-administered questionnaires. However, these data can suffer from a variety of problems—e.g., no opinion responding, non-differentiation, acquiescence—in part because respondents may not pay attention as they answer a questionnaire (Krosnick 1991; Berinsky, Margolis, and Sances 2014). This study examines the effectiveness of warnings as a method for improving data quality in self-administered surveys. Generally speaking, warnings encourage respondents to follow instructions or to think carefully (Krosnick 2000). There is evidence that such messages can be effective (Huang et al. 2012), but warnings may increase social desirability pressures. Moreover, this effect may be concentrated among particular respondents, such as the highly educated, because of their greater overall tendency to offer socially desirable responses (Silver, Anderson, and Abramson 1986).