Smart MRAG: Hybrid Retrieval-Augmented Multimodal Q&A Technique

Generative AI models and advanced techniques like retrieval augmented generation (RAG) have found use-cases in several application areas requiring accurate question-answering. This study proposed a new technique named Smart multimodal RAG (Smart MRAG) which combines RAG with multimodal models to provide accurate answers by retrieving data from PDFs. The data are not limited to just text, Smart MRAG excels in retrieving data from tables and figures as well. Multimodal models are good at retrieving data from two‑dimensional structures such as tables, while RAG excels at retrieving relevant data required for answering questions correctly. Smart MRAG technique proposed in this study is a hybrid approach to achieve not only accurate results but also achieves these results in a cost-effective way. This study compares the performance of three techniques: pure text-based RAG, pure multimodal RAG, and Smart MRAG. Our results show that Smart MRAG significantly outperforms text-based RAG and pure multimodal RAG.