Faithfulness Evaluation of an Uzbek-Language Rag System: A Pilot Study and Lessons on LLM-Judge Pitfalls

Retrieval-augmented generation (RAG) systems can still produce statements that the retrieved source does not support, yet for Uzbek this problem has hardly been studied. This pilot study evaluates Siyrat RAG, an Uzbek-language question-answering system built on three Uzbek books about the life of Prophet Muhammad. We ran 18 questions through two language models (Qwen3.8-27B and GPT-OSS-20B) with an identical retrieved context and scored the answers with an LLM judge. We also report four defects in our first benchmark: failed API requests counted as zero scores, truncated reasoning text sent to the judge, a judge that never saw the retrieved passage, and a judge drawn from the evaluated models. Together they produced a misleading result. After correcting them, we checked all 36 answers by hand against the context that the model had received (the top 10 chunks, first 600 characters of each). Only 2 of 36 answers contained unsupported claims. In 16 of 36 cases the answer was absent from this context; in 14 of these the model correctly refused and in 2 it invented an answer. The LLM judge flagged 8 answers; the annotator confirmed 2 of these flags, a precision of 25% for the flags, which is not the overall accuracy of the judge. The results are preliminary: the 36 answers come from only 18 questions and are not independent trials, and there was a single annotator. They suggest that gaps in the context passed to the model account for most failures and that automatic judges need human validation for Uzbek.