Direct comparison tests for effective evaluation of text-to-speech with a small number of listeners

In this article, we contrast different subjective tests for the evaluation of text-to-speech systems. We compare results obtained via MOS, a five-point absolute rating test, to two direct comparison tests: the AB test and the Best Worst Scaling (BWS) test. In an AB test, participants select their preferred stimulus from two options, while in a BWS test, they select the best and worst from multiple stimuli. We evaluate these three tests in terms of session duration, participant consistency, number of significant differences detected, and system ranking. Additionally, we estimate the impact of number of participants and test duration in their performance. To make this comparison, we conducted three separate listening tests (MOS, AB and BWS) and analysed results derived from all collected ratings, as well as results derived from a reduced sets of listeners and stimuli. Our findings indicate that BWS tests provide more user ratings within the same amount of time. We also found that participants are more consistent with each other and themselves in the BWS test. Our results also show that the BWS test reveals more significant differences than both AB and MOS, especially with fewer participants or with shorter tests. In particular, we found that a BWS test with as few as 10 participants reveals as many differences as a MOS test with 20.