Refining the evaluation of speech synthesis: A summary of the Blizzard Challenge 2023

The Blizzard Challenge has benchmarked progress in Text-to-Speech (TTS) since 2005. The Challenge has seen important milestones passed, with results suggesting that synthetic speech was indistinguishable from natural speech in terms of intelligibility in 2021 and that by that same year it was perhaps even indistinguishable in naturalness. The high quality of synthetic speech generated by the latest TTS systems has thus revealed limitations with ITU-T P.800.1 Mean Opinion Score (MOS) in detecting the remaining differences between synthetic and natural speech. Yet, it was the only method used in previous Challenges and is still the most popular method in the field for speech synthesis evaluation. In the 2023 Challenge, we addressed observed limitations of past Challenges by incorporating state-of-the-art speech synthesis evaluation techniques to refine the evaluation of speech quality, speaker similarity and intelligibility. For speech quality, a relative comparison of the systems receiving the best MOS was able to discover a greater number of significant differences between systems. Regarding speaker similarity, we demonstrated that there is a strong bias depending on whether the listeners are familiar with the target voice or not. As for intelligibility, the evaluation of language-specific phenomena, such as the pronunciation of homographs, better highlighted system limits compared to global transcription tasks of synthesised utterances. In addition to reporting results for the 18 entries to the 2023 Challenge, we extend the results analysis to type of TTS module to provide some insights on the most recent advances in model design. Overall, this year’s results demonstrate the need for a shift towards new methods for refining TTS evaluation to shed light on increasingly smaller and localised differences between synthesised and natural speech. • We refined the assessment of speech quality, speaker similarity and intelligibility in a large-scale benchmark of 18 text-to-speech systems (Blizzard Challenge 2023). • While global Mean Opinion Score for speech quality has reached its limits for discriminating the best generated speech signals from natural speech, a fine-grained relative comparison between speech generated by the best systems and natural speech allowed to discover the remaining differences. • Speaker similarity evaluation showed that average listeners are not able to perfectly recognise the natural voice given as a reference, which calls into question the speaker similarity evaluation paradigms for synthetic speech. • The evaluation of specific local events, such as the correct pronunciation of homographs, highlighted differences between systems that a generic transcription task cannot, which also motivates the use of fine-grained evaluation for intelligibility assessment.