Question characteristics and their effect on measurement quality: Lessons learned from the European Social Survey

The main challenge for questionnaire designers is to create survey measurement instruments that capture the true responses from the population, minimizing the measurement errors. While there is expert knowledge and standard procedures, there is still not enough empirical evidence for the influence of all the different possibilities questionnaire designers have when creating a survey item. In this study we use the data from seven rounds of Multitrait-Multimethod (MTMM) experiments ran in the European Social Survey to shed light on the effect of response scale characteristics (number of response categories; presentation on visual aid, horizontal or vertical presentation; labels, fixed reference point; order of response categories) and characteristics of the request for an answer (formulation, type and use of gradation) on both the True Score reliability and True Score validity of survey questions. Our data set includes more than 3000 survey items, their question characteristics as well as their True Score reliability and True Score validity estimates from 22 MTMM experiments that have been fielded in 28 languages. Our analysis accounts for the hierarchical structure of the data where survey items are nested in experiments and languages. Applying regression methods to the experimental data set allows us to estimate causal effects of each question characteristic while controlling for the others.