Evaluation of RAG-based LLMs: A case study in the insurgency knowledge domain

Retrieval Augmented Generation is a technological response to the challenges that LLMs face of being limited by a knowledge cut-off date and of being trained only on public data. The evolving stack of Naïve, Advanced and Modular RAG has tried to alleviate some of these shortcomings; but their better performance on the scores of accuracy and relevancy has faced an inevitable trade-off in terms of greater latency, higher computational complexity and heavier processing costs. Across all these stages of RAG evolution, challenges to RAG evaluation persist, owing to lack of standardized metrics, reliance on LLMs as both generator and judge of its own output, and the domain-agnostic nature of most benchmarks that may not thus capture the performance of a RAG system in a specific domain . A plethora of performance metrics makes inter-model comparison difficult when two or more models outperform each other on a different metric. The present paper offers a novel game theoretic framework that pins down the evaluation of a RAG system to a single value (that subsumes a range of performance metrics, including Correctness, Faithfulness, Relevancy and others) thereby facilitating inter-model comparison. The game-theoretic framework also deflates the performance of a model to those parameters where its results are poorest: we empirically demonstrate a RAG-based model scoring over 90% in each parameter of Faithfulness, Relevancy and Similarity but being given an overall performance score of just 59% owing to its poor performance in Correctness. This calibration is based on a zero-sum game theoretic model; and it also obeys the adage that the strength of a model is its weakest link. The paper also introduces a faster algorithm for computing mixed strategy zero sum games (to arrive at the performance value of the model) than standard linear programming solvers.