Prompt Engineering for Mitigating Object Count Mismatch in Stable Diffusion Model

Key Points

• The study analyzes whether prompt engineering can mitigate object count mismatches in the Stable Diffusion model without modifying the model itself.
• Compared three types of prompts: basic quantity explicit prompts, human-designed structured prompts, and LLM-based self-refine prompts.
• Text-to-image generation was performed using Stable Diffusion 1.5 and evaluated with YOLOv8.
• Count reproduction performance was measured using metrics: count accuracy, mean absolute error, under-rate, and over-rate.
• Structured prompts showed some improvement over basic quantity explicit prompts.
• Self-refine prompts significantly improved count accuracy compared to other types.
• The approach using prompts alone cannot completely control object count errors, indicating fundamental limitations.