Graph retrieval-augmented generation for legal academic viewpoint question answering: knowledge-graph retrieval, evaluation, and hallucination analysis

Large language models (LLMs) have a common problem of hallucination in legal academic viewpoint question answering, which limits its reliability in legal research assistance. To alleviate this problem, this study uses Chinese core legal journal literature as the data source to construct a graph retrieval augmented generation (GraphRAG) framework for legal academic viewpoint question answering in this field and systematically compares the performance of two strategies, namely Vector and Cypher retrieval, on three LLMs. The experimental results show that with the Cypher retrieval strategy under the neo4j_graphrag framework, the unanswerable rate of gpt-4-turbo decreased to 3.41% and the BERTScore-F1_avg reached 80.58%, which provides preliminary evidence for the effectiveness of structured knowledge graph retrieval in improving the answer quality. Through example analysis and quantification, this study summarized five GraphRAG-specific hallucination types, revealing that GraphRAG alleviates traditional knowledge-deficiency hallucination while shifting the source of hallucinations from the blind fabrication of the model to the collaborative retrieval and generation process, and that certain differences exist in the distribution of hallucination types under different retrieval strategies. This study focuses on the development and optimization of a legal academic question and answer system under the theme “Legal Regulation of AI Training Data,” within a known and closed corpus, and provides a reference for the study of possible hallucination generated by the GraphRAG framework in this context.