Answers are grounded in this paper and cite their sources.
Image captioning is a classic vision and language generation task that aims to generate a descriptive sentence for an input image. Inspired by advances in computer vision and natural language processing, numerous approaches have been proposed to solve this problem. However, current methods often lack sufficient contextual information to comprehensively describe the corresponding images, failing to fully capture their key features or essential details. Additionally, training data sets annotated with captions typically offer only basic descriptions of the images, limiting the ability of models to capture the full breadth of visual content. In this work, we present the VQA-LLM Fusion Caption Generator (VLFuseCap), a novel framework that addresses these limitations by seamlessly integrating visual question answering (VQA) with large language models (LLM). To fill in the gaps in the current datasets, we utilize GPT-4 to generate the necessary data. Using the complementary strengths of VQA in visual understanding and the robust capabilities of LLM in natural language processing, our approach can generate detailed and precise descriptions. We demonstrated the effectiveness of our approach through comprehensive evaluation. Our framework has broad applications, particularly in robotics, where it enhances language comprehension capabilities and fosters new avenues for personalizing machine interactions. The code is available at https://github.com/galatolofederico/llm-chain-captioning .
No takes yet. Share an insight, caveat, or question.
Galatolo et al. (2026) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: