Reducing the Bias From Probability and Nonprobability Online Panels by Excluding Satisficers

Abstract Over the last 15 years or so, many studies have compared the accuracy of estimates produced when the same questionnaire is administered to samples generated using probability and nonprobability sampling methods. Most studies conclude that estimates from probability sample surveys are more accurate than estimates from nonprobability sample surveys. In trying to understand why probability sample surveys generally produce more accurate estimates than nonprobability sample surveys, most research has focused on the differences in survey samples reached as a result of the sampling mechanisms used by each approach, and looked to various adjustment (weighting) approaches to bring the nonprobability sample estimates closer to the comparative probability sample estimates and independent benchmarks. Less common, however, are comparative studies that focus on measurement error. This study compares estimates generated from five Australian online panels; one probability-based online panel and four nonprobability online panels. We use widely implemented techniques to identify instances of survey satisficing—the phenomenon of survey respondents providing lower quality responses to reduce cognitive effort. When comparing the level of satisficing across panels, a higher proportion of respondents in the nonprobability panels give responses consistent with satisficing than in the probability panel. We then show that excluding these suboptimal responses from our analysis dataset, before weighting, results in a greater bias reduction across the nonprobability panels than weighting alone, and that this is the case across all four nonprobability panels. In contrast, very little change to bias is observed after applying the same exclusions to the probability panel. We end with a call for others to undertake similar research in the hope that this approach might become part of the toolkit of techniques being developed to reduce the bias in survey estimates generated from nonprobability online samples.