Evaluation of Knowledge Graph Integration in LLM-Based Financial Chatbots

This study is a comparison of two personal finance chatbot models: LLM only and Knowledge Graph enhanced. They were compared in two criteria: accuracy of answers and response time. The KG was created from the financial transaction dataset. A chatbot architecture consists of two prompts to LLM: the first prompt asks LLM to generate a code that retrieves the context to answer the user’s question, and the second prompt queries LLM to frame a response using the context. The chatbots were assessed in accuracy by answering 64 personal finance questions. Results show that the LLM-only chatbot is more accurate than the KG-enhanced chatbot. It is the opposite in response time, the KG-enhanced chatbot is faster than LLM-only chatbot. The research aims to add knowledge to the research field of combining Knowledge Graphs with LLMs and to help practitioners by presenting the benefits and downsides of KG integration.