Answers are grounded in this paper and cite their sources.
We describe an approach to crowdsource the evaluation of TTS systems by preference tests and report on lessons learnt from running 127 real-life crowdsourced tests. We show that at least one type of cheating becomes more prevalent over time if left unchecked and develop metrics to exclude cheaters. We demonstrate that their exclusion improves test outcomes. Index Terms: TTS, speech synthesis, listening test, preference test, crowdsourcing, cheating
No takes yet. Share an insight, caveat, or question.
Buchholz et al. (2011) studied this question.