LightEdit: Textual Image Editing With Lightweight Stable Diffusion

This is the authors' abstract. We don't add key points for this paper.

Text-based image editing using diffusion models has shown remarkable progress, but most approaches rely on large-scale models (Stable Diffusion Models - SDMs) that require significant computational resources. In this work, we propose a lightweight diffusion-based model for textual image editing that balances efficiency and performance. We introduce a specialized U-Net architecture with channel weight cross-attention mechanisms. The model architecture is optimized for image editing quality (preserving the original image structure while editing the region specified by the prompt) while reducing parameter count. We leverage knowledge distillation with multiple stage-dependent teacher models to inherit the knowledge from a large-scale pre-trained editing model. Our model achieves better results than some other state-of-the-art methods for almost all evaluation metrics on the InstructPix2Pix dataset. Moreover, our proposed UNet is approximately 2×, and 5× (170M parameters) smaller than the model with BK-SDM and Stable Diffusion backbone, respectively. Human evaluations also confirm the superior visual quality of our edited outputs. Code is available at: https://github.com/thuyvuphuong/LightEdit-Textual-Image-Editing-with-Lightweight-Stable-Diffusion-Repository.