LayerDiffusion: High-Fidelity Layered Virtual Try-On via Semantic-Aware Diffusion

Image-based Virtual Try-On (VTON) aims to generate virtual try-on results by transferring input garment images onto target person images. Current algorithms have achieved high generalization and quality on public datasets including VITON and DressCode. However, current algorithms primarily process images at the pixel level while neglecting the semantic information of garments. Moreover, layered outerwear, as an essential aspect of try-on, has been largely overlooked. Layering increases the difficulty of precise garment control due to the need to consider inner garments. Furthermore, the task-specific nature of current methods prevents them from capturing layering characteristics in try-on. To address these limitations, we propose LayerDiffusion, a novel framework that enhances both generalization capability and semantic controllability for virtual try-on generation. Our approach makes three key contributions. First, we construct LayerDataset, a specialized dataset focusing on diverse outerwear garments with balanced gender representation and multi-view captures. Second, we integrate a pretrained image encoder to capture rich semantic garment information, combined with multi-conditional masking and adapter fine-tuning to enable flexible layering control. Third, we introduce a joint training strategy on both try-on and try-off tasks, which substantially improves model generalization by learning bidirectional garment-body correlations. Extensive experiments demonstrate that LayerDiffusion significantly outperforms existing state-of-the-art methods on public benchmarks (VITON-HD and DressCode) across multiple metrics including LPIPS, SSIM, FID, KID, and CLIP-based scores. Our method excels in both single-garment and layered try-on scenarios while naturally supporting try-off capabilities, offering superior semantic preservation and detail fidelity compared to existing approaches.