Convenient but biased? The reliability of convenience samples in research about attitudes toward climate change

Ideally, research in the social and behavioral sciences would be conducted using samples that perfectly represent the population of interest. However, it is often not feasible for researchers to collect representative samples, forcing them to rely on convenience samples. While some researchers have found that convenience samples can produce comparable results to population- based samples, others have found evidence of biased results. This suggests that differences in the composition of study samples (e.g., demographic, ideological) can sometimes influence study results, leading to contradictions in the literature. To test this, we conducted four identical experiments across three platforms (Amazon Mechanical Turk, TurkPrime Panels, and Facebook) using two different study designs (mixed factorial and between-subjects). We tested a finding known to be robust: communicating the descriptive norm that most climate scientists are convinced that human-caused climate change is happening leads respondents to update their own beliefs about the issue. The results suggest significant variability in effect sizes depending on the sample source and study design, with effect sizes ranging from near-zero to large. Additionally, we make novel use of rake weighting to adjust for inter-sample differences (demographics, ideology, message familiarity) to determine whether differing results from identical experiments are due to underlying sample differences or to the unreliability of the effects. We discuss the implications these results have for experimental research, especially around politicized issues such as climate change.