Time after time, the Best Worst Scaling method is more reliable than ranking

Two popular methods of preference elicitation are rankings and Best-Worst Scaling (BWS). Rankings, while simple and widely adopted, can be burdensome with larger item sets and fail to capture indifference between options that are neither loved nor hated. Best-Worst Scaling is a survey method that sidesteps the set size problem by capitalising on people’s natural capacity to identify preferences at the extremes. Across three experiments, our primary finding is that elicited preferences for ranking and BWS methods align, and that BWS methods provide additional resolution to resolve the indifference between middling options where rankings can struggle as well as the relative importance of each option. Moreover, we show that BWS methods exhibit greater test-retest reliability compared to rankings, even over time frames as short as minutes. Taken together, our results privilege BWS as a reliable and readily accessible alternative to ranking methods for preference elicitation.