Semantic Driven Stylized Image Captioning For Artworks

Most existing image captioning methods have shown impressive performance on real-life images, but few of them explore to generate stylized captions on artificial images like artworks. There are two challenges in this task: firstly, modern techniques usually extract fine-grained visual features by using pretrained object detectors, which perform poorly on artworks due to the domain gap, further limits the whole model's performance. Secondly, diverse emotional inclinations may result in varying attention to semantics in image, making it challenging to establish a clear correlation between emotion and visual input. To address these issues, we propose our object detector-free Semantic-Driven Stylized Image Captioning Networks (SD-Net). To be specific, we initially utilize an encoder to acquire visual tokens on CLIP image features, then enhance it with an additional emotion token. After that, a semantic retrieval module is employed to directly reconstruct emotion-related semantic words. Finally, we integrate the visual tokens and semantic words into a sentence decoder to generate the stylized caption. Our experimental results demonstrate that our approach achieves competitive performance on Artemis dataset in various benchmarks.