Enhancing the Efficiency of Large Multimodal Models for Document Understanding

Effective document understanding is crucial for enhancing the capability of production-scale retrieval-augmented generation (RAG) systems, yet existing multimodal large language models (MLLMs) often struggle with accurately extracting and interpreting numerical and contextual information from charts. In this work, we propose a practical method that improves document understanding in RAG systems through domain-adaptive MLLM fine-tuning and systematic generation prompt optimization. We construct a domain-specific dataset tailored to business needs and employ advanced prompt engineering techniques to maximize MLLM performance. Furthermore, we develop a customized evaluation framework designed for document understanding and conduct extensive comparisons between state-of-the-art MLLMs and our fine-tuned models across different benchmarks. Experimental results demonstrate that our optimized model outperforms third-party baselines, improving accuracy by 7.6% relative in RAG-based question answering. Furthermore, the integration of our approach into a production-scale RAG system highlights its practical applicability in real-world scenarios. Our findings provide insights for advancing research on multimodal RAG and document-based reasoning tasks.