Testing Methods for Retrieval Augmented Generation Models in Russian

The application of generative models that use information retrieval, or the retrieval augmented generation approach, where a search engine retrieves relevant documents at the user’s request and the generative model produces an answer by combining the retrieved context with its internal knowledge, overcomes the limitations of large language models, such as knowledge obsolescence and the inability to handle dynamically changing data. The performance of such systems directly depends on the search quality, the context structure, and the wording of the instructions, but a comprehensive evaluation of this approach is a challenging task at present, not only due to the low number of suitable datasets and differences in the requirements for systems, but also because there are no standard metrics to measure the accuracy of the response in relation to the provided context. In this work, we conducted a comparative study of six language models (DeepSeek, Llama, Mistral, Qwen, RuAdapt, and YandexGPT) in a question-answering task using the retrieval augmented generation approach. The experiments were performed on four Russian-language datasets—XQuAD, TyDi QA, RuBQ, and SberQuAD—which were converted to a unified format suitable for the task. Various strategies for adding and arranging fragments in the context provided to the model were considered, and the evaluation of the models using the metrics of context relevance, utilization, completeness, adherence, and exact matching allowed us to identify the limitations of generative models in extracting information from relevant context.