Pretesting Survey Questions Via Web Probing – Does it Produce Similar Results to Face-to-Face Cognitive Interviewing?

Asking probing questions via web probing has recently been advocated as a promising method for evaluating survey questions. In comparison to standard face-to-face (f2f) cognitive interviewing, the increasing availability of internet non-probability panels allows for recruiting respondents in a quicker and more cost-effective way and a realization of larger sample sizes. In the present study, we examine whether web probing is a potential alternative to standard cognitive interviewing, in particular: Does web probing produce similar results as f2f cognitive interviewing with regard to the problems detected and the item revisions suggested? The study compares the findings of 508 respondents drawn from a non-probability online panel who completed an online survey including four items from the International Social Survey Programme 2013 and 2014 with the results obtained via f2f cognitive interviewing with 20 participants. Findings indicate that web probing and cognitive interviewing detect very similar problems and lead to the same suggestions for item revisions. However, web probing itself has some limitations. Practical implementations and directions for future research are discussed.