SD-Prompt: Learnable and Adaptive Prompts for Enhancing Subject-Driven Text-to-Image Synthesis

Large-scale pre-trained text-to-image models have made incredible progress recently. When synthesizing the appearance of subjects in given texts, existing works fine-tune the pre-trained model or use manual prompts to improve the quality of images. However, fine-tuning requires storing parameters for each subject, which results in expensive computational costs, and manual prompts need the artificial selection of different subjects, which brings subjective uncertainties. To enhance the quality of synthetic images and reduce computational costs, we introduce theSubject-DrivenPrompt(SD-Prompt). Specifically, we propose a prompt generation module to reduce computational costs through learnable prefixes. Furthermore, we propose a text-adaptive prompt selection module that evaluates the feature similarity between texts and prompts to automatically accommodate the need for subject-driven image generation. To explore the performance of prompts in complex scenarios, we also collect the CC3M-S5C dataset from CC3M dataset. Experimental results show that our work outperforms other methods both in image fidelity and text-image semantic consistency.