Comparative study improves visual question answering accuracy in resource-constrained environments, suggesting new protocols.
Key Points
This research aims to evaluate the effectiveness of Retrieval-Augmented Generation (RAG) in enhancing small language models for visual question answering.
Developed a light multimodal RAG (MM-RAG) pipeline on consumer-grade hardware.
Compared two small language models: TinyLlama (1.1B) and Qwen 2.5 (3B).
Implemented a post-processing protocol to improve output accuracy.
Achieved a significant accuracy increase of 13%-16% with RAG compared to zero-shot baselines.
Identified verbosity failure in instruction-tuned small language models that leads to low evaluation scores.
Restored Qwen's accuracy from 8% to 52.6% using the developed post-processing protocol.