Retrieval Free Context Augmentation for Image Captioning: Learning from Self-Generated Beam Hypotheses

Vision-language models (VLMs) generate image captions by grounding visual perception in semantically coherent natural language, a process whose quality is largely determined by the supervision signal used during training rather than by model capacity alone. Existing Retrieval-Augmented Generation (RAG), approaches rely exclusively on fixed human-annotated captions, leaving unex-ploited the rich, diverse hypotheses that the model itself generates during inference. To address this limitation, we introduce CAG-Cap, a retrieval-free caption-refinement framework inspired by the offline caching principle of 1 Cache-Augmented Generation (CAG). Unlike conventional CAG, where cached knowledge is injected into the inference context, CAG-Cap uses cached information exclusively as auxiliary sequence-level supervision during training, with no additional inference-time parameters or retrieval operations. CAG-Cap operates in two stages. In Stage A, a baseline captioner comprising a frozen SigLIP 2-B/16 vision encoder and a four-layer Transformer decoder is trained under next-token prediction to convergence, a caption cache is then constructed by running beam search over the training set and retaining the top-k hypotheses per image as self-generated pseudo-labels. In Stage B, the model is fine-tuned under a dual-loss objective that combines ground-truth negative log-likelihood (NLL) supervision with a CAG loss, defined as the mean NLL over all k cached hypotheses, weighted by a tunable coefficient λ CAG. Experiments on MS COCO 2017 and Flickr30k demonstrate the effectiveness of CAG-Cap. The optimal configuration on MS COCO 2017 achieves a CIDEr of 2.2106, a BLEU-4 of 0.3902, and a SPICE of 0.4115, on Flickr30k, it attains a BLEU-4 of 0.3218 and a METEOR of 0.4990. CAG-Cap establishes a principled, retrieval-free alternative to RAG for training-time caption supervision, offering a scalable pathway toward self-improving VLMs. The source code is accessible: https://github.com/abimouloud/CAG-Cap