Fine-tuning strategies for large-scale face-conditioned text-to-speech

Speech-conditioned text-to-speech (TTS) models, which generate speech from text and reference speech, have been successful due to large-scale datasets and parameter size. Similarly, scaling up face-conditioned TTS models, which generate speech from text and facial images, is expected to enhance both speech quality and speaker similarity. However, the effectiveness of such models is hindered by the scarcity and low quality of audio-visual data, which often contain noise in either the speech or the images. Previous work has leveraged a large-scale speech-conditioned TTS model as a pre-trained model for fine-tuning with an audio-visual dataset. This approach compensates for the limited quantity and quality of such data, while achieving speaker similarity comparable to that of speech-conditioned models. While fine-tuning the TTS model improves speaker similarity, it degrades intelligibility, which is likely influenced by noise in the audio-visual data or by the modality gap between the conditioning inputs used during training and inference. To address this issue, we investigate fine-tuning strategies for face-conditioned TTS by randomly mixing audio-only and audio-visual data during fine-tuning, and by evaluating whether to use the audio-only or the audio-visual dataset during each training stage. In addition, we conduct comparative experiments to evaluate the effectiveness of these strategies.