Context-aware retrieval-augmented generation framework for visual question answering in medical imagery

Medical Visual Question Answering (VQA) has underwent significant advancements through the adoption of large vision-language models (VLMs), which typically require fine-tuning for downstream tasks. However, fine-tuning can pose challenges, including high computational demands and potential overfitting risks that can compromise generalisation, especially when training data are limited. In this paper, we introduce an alternative approach termed RAG-MedVQA, a retrieval-augmented framework that integrates retrieval and generation components. Our framework adopts a modular design that decouples the retrieval and generation components, enabling flexible integration with a variety of model choices. In the retrieval phase, we leverage the BioMedCLIP retriever to compute dense image embeddings and retrieve the most similar images from the training set to the query image along with their corresponding diagnostic dialogues. In the generation phase, the query image, its associated question, and the retrieved context are processed by Qwen2VL-7B, a multi-image capable vision-language model that fuses clinical content from the exemplars to synthesise the final answer. More specifically, we form a structured prompt to guide Qwen2VL-7B effectively during response generation. The experimental results show that RAG-MedVQA achieves competitive performance compared to conventional fine-tuning approaches.