Attribute-Conditioned Attention Scaling for Text-to-Image Diffusion Models

Many text-to-image diffusion models have attained significant success in the generation of images from textual prompts; however, they still face some challenges, such as limited fine-grained control over different semantic attributes. To overcome this issue, this study proposes an Attribute-Conditioned Attention Scaling (ACAS), which modulates the cross-attention layers of the UNet model using attribute-related scaling factors. These scaling factors are assigned to the attention maps that allow selective enhancement of features in the ACAS without retraining. Moreover, this model allows precise control over generated images without retraining of the base model at the inference level, along with preservation. For experiments, 30 diverse prompts along with eight descriptive attributes are used to inspect the multi-attribute controllability of the proposed model. Different evaluation metrics, such as CLIP, LPIPS, and Inception Score (IS), are used to quantitatively evaluate the proposed model. Experimental results prove that the proposed model ACAS obtains competitive results with an LPIPS of 0.75, CLIP score of 0.316, IS of 3.98, and a minimal 8.47 seconds generation time. Furthermore, a comparative analysis of the ACAS model with similar baseline methods is performed, and the comparison shows that the ACAS improves attribute controllability without adding extra computational cost. Overall, this model bridges the gap between fine-grained attribute control and prompt-based guidance in the latest diffusion models.