KnowRAG: A Zero-Shot Diagnostic Analysis of Knowledge Base Coverage in Scientific Retrieval-Augmented Generation

The "hallucination" problem in Large Language Models (LLMs) remains an unresolved hurdle for scientific researchers who require precise, grounded evidence. While Retrieval-Augmented Generation (RAG) aims to mitigate these errors, standard systems are often unoptimized for the structural complexities of scientific papers. We introduce KnowRAG, a zero-shot RAG pipeline specifically designed for scientific applications. Using a novel "LLM-as-a-Judge" diagnostic framework, we evaluated KnowRAG against a standalone GPT-3.5-Turbo baseline across four specialized Q&A Test Sets. Our results demonstrate that KnowRAG significantly improves factual accuracy over the baseline. More importantly, diagnostic analysis reveals that the vast majority of errors (over 46%) stem from Knowledge Base Coverage (knowledge gaps), while generation failures remain negligible at 4%. These findings suggest that retrieval and generation capabilities are no longer the primary bottlenecks in the scientific domain. Instead, this diagnostic analysis advocates for a paradigm shift from model-centric research toward expert data engineering as the definitive path to trustworthy AI. By repurposing the LLM-as-a-Judge framework as a diagnostic instrument rather than a mere performance metric, we move RAG evaluation beyond aggregate scoring toward actionable, evidence-based systemic diagnosis.