Learn to Prompt: Soft-Prompting Adaptive Attention for Image Captioning

Attention-based neural network encoder-decoder frameworks are commonly used for image captioning. However, they may not fully comprehend "what to look at" and "how to describe it". This paper introduces the "Soft-Prompting Adaptive Attention" model, which improves image captioning performance by simultaneously integrating learnable soft prompts into the encoder and decoder. The introduced prompts are designed to learn complex and important visual characteristics, which can enhances the model's understanding of complex image content and control over caption generation. As a result, the model exhibits exceptional performance on the COCO benchmark dataset, producing more precise and detailed captions, especially for complex and varied images. The paper also includes comprehensive experiments that confirm the efficacy of the soft prompting mechanism in image captioning. The code and pre-trained models are available at https://github.com/ovvo20/soft-prompt-adaptive.git.