Answers are grounded in this paper and cite their sources.
Experimental study examines response category impact on survey behavior, revealing modest reliability improvements.
This study investigates how the number and labeling of response categories in survey scales affect respondent behavior, psychometric properties, and substantive conclusions. Using data from a web-based survey experiment with over 2,800 participants, we randomly assigned respondents to scales that varied in the number of response options and labeling formats. Engagement, response style tendencies, reliability and convergent validity were assessed through a combination of process data (e.g., response times, cursor movements), self-reports, and psychometric modeling. Our findings suggest that scales with a greater number of response categories tend to exhibit higher reliability, although the improvements are modest and depend on the specific scale. While scales with more response options required longer completion times and prompted more complex response behavior, the number of response categories did not influence substantive conclusions in simple regression models.
No takes yet. Share an insight, caveat, or question.
Pokropek et al. (2025) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: