Is Retrieval-Augmented Generation All You Need? Investigating Structured External Memory to Enhance Large Language Models’ Generation for Math Learning

Interactive question-answering (QA) with a tutor is an effective learning method for middle school math students. The flexibility and emergent capabilities of generative large language models (LLMs) have sparked significant interest in automating this process to deepen conceptual understanding of mathematical learning. However, the issue of LLM hallucination has been well-documented in the research community. LLMs' responses to math problems can be inaccurate or misaligned with educational contexts, such as school curricula. A potential mitigating solution is retrieval-augmented generation (RAG), which involves integrating external knowledge sources into LLM prompts to enhance response quality. Although traditional RAG with vector embeddings has shown promise in improving generation quality, there are limitations regarding its information retrieval scalability, granularity, and explainability. One potential method to further improve traditional RAG is using a knowledge graph (KG). In this paper, we design a method to construct a KG with LLM, retrieve contextual prompts from high-quality math teaching resources via the KG, and then enhance the prompts to generate answers to real student questions about mathematical concepts. To evaluate the effect of retrieved information in RAG, we designed three levels of guidance prompts corresponding to three degrees of groundedness: high, low, and none. We evaluate from two perspectives: the semantic similarity between the retrieved information context and the generated text context, and conduct an experimental study to investigate the impact of our KG-enhanced QA system on student learning gains. The study found that compared to baseline methods, the KG-based RAG approach can enhance the groundedness and faithfulness of both the retrieved documents and the generated context. Additionally, low guidance can improve student learning gains. Our technical framework and analysis results contribute to advancing the application of LLMs in educational technology.