DSUSING: Dual Scale U-Nets for Singing Voice Synthesis

Singing voice synthesis (SVS) is a task of generating acoustic features such as mel-spectrograms or spectrograms from music scores and lyrics. In this paper, We present DSUSing, which is composed of spatial and frequency U-nets [1] that predict strips of mel-spectrogram in auto-regressive manner. The feature of these strips are integrated later with the proposed fusion layer. By introducing two domains of U-nets, our method demonstrates an enhanced comprehension of the temporal and frequency relationship through learning different domains of segments in the mel-spectrogram. Our architecture is motivated by [2], a Generative Adversarial Network (GAN) [3] based Korean singing voice synthesis model and outperforms it in both subjective and objective evaluations. The experimental results on the public Children's Song Dataset (CSD) [4] shows that our proposed model synthesizes more natural and higher quality singing voices.