Exploring Controllable Ability through Text-to-Image Diffusion Model for Painting-Style Design

This is the authors' abstract. We don't add key points for this paper.

Painting style and creativity are fundamental to art, defining a work’s uniqueness and expression. They embody an artist’s personality, convey emotions, establish identity, and shape audience perception, adding depth and value to the work. Recently, diffusion models have gained popularity in art design, animation, and gaming, especially in original painting creation, poster design, and visual identity development. Traditional creative processes, while demanding unique imagination and creativity, often face challenges such as slow innovation, high reliance on manual efforts, high costs, and limitations in scalability. Consequently, exploring painting-style creative design through deep learning has emerged as a promising trend. In this paper, we introduce a novel network architecture, the Painting-Style Design Assistant Network (PDANet), for style transformation. To support this, we curated the Painting-42 dataset, comprising 4,055 works by 42 renowned Chinese painters. PDANet leverages this dataset to capture the aesthetic intricacies of Chinese painting, offering rich design references. Furthermore, we propose a lightweight Identity-Net for large-scale text-to-image (T2I) models, which aligns internal knowledge with external control signals, thereby enhancing the capabilities of existing T2I models. The trainable Identity-Net feeds image prompts into the U-Net encoder to generate diverse and stable images. Both quantitative and qualitative analyses demonstrate that our approach outperforms current methods, delivering high-quality generated content with wide-ranging applicability.