What type of study is this?

This is a Experimental Study study.

October 16, 2025Open Access

REF-VC: Robust, Expressive and Fast Zero-Shot Voice Conversion with Diffusion Transformers

Key Points

REF-VC enhances noise robustness while maintaining expressiveness in voice conversion tasks, outperforming traditional methods.
Experimental results show that REF-VC performs better in zero-shot scenarios, achieving significant improvements in noisy conditions.
Key innovations such as random erasing and integration of Shortcut Models streamline the conversion process, reducing steps substantially.
REF-VC also supports singing voice conversion, broadening its application beyond standard voice scenarios.

Abstract

In real-world voice conversion applications, environmental noise in source speech and user demands for expressive output pose critical challenges. Traditional ASR-based methods ensure noise robustness but suppress prosody richness, while SSL-based models improve expressiveness but suffer from timbre leakage and noise sensitivity. This paper proposes REF-VC, a noise-robust expressive voice conversion system. Key innovations include: (1) A random erasing strategy to mitigate the information redundancy inherent in SSL features, enhancing noise robustness and expressiveness; (2) Implicit alignment inspired by E2TTS to suppress non-essential feature reconstruction; (3) Integration of Shortcut Models to accelerate flow matching inference, significantly reducing to 4 steps. Experimental results demonstrate that REF-VC outperforms baselines such as Seed-VC in zero-shot scenarios on the noisy set, while also performing comparably to Seed-VC on the clean set. In addition, REF-VC can be compatible with singing voice conversion within one model.

REF-VC: Robust, Expressive and Fast Zero-Shot Voice Conversion with Diffusion Transformers

Key Points

Abstract

Cite This Study

Also Consider

Also Consider