Generating diverse and natural text-to-speech samples using a quantized fine-grained VAE and auto-regressive prosody prior

Recent neural text-to-speech (TTS) models with fine-grained latent features precise control of the prosody of synthesized speech. Such models incorporate a fine-grained variational autoencoder (VAE) structure, latent features at each input token (e. g. , phonemes). However, samples with the standard VAE prior often results in unnatural and speech, with dramatic prosodic variation between tokens. This proposes a sequential prior in a discrete latent space which can generate naturally sounding samples. This is accomplished by discretizing the features using vector quantization (VQ), and separately training an (AR) prior model over the result. We evaluate the approach using tests, objective metrics of automatic speech recognition (ASR), and measurements of prosody attributes. Experimental results show the proposed model significantly improves the naturalness in random sample. Furthermore, initial experiments demonstrate that randomly sampling the proposed model can be used as data augmentation to improve the ASR.