When structure predicts hallucination: Aligning LLMs with knowledge graph features

Large Language Models (LLMs) have demonstrated remarkable factual accuracy in producing human-like and AI-generated texts across a wide range of natural language tasks, including question answering. Despite these advances, their tendency to hallucinate and produce fabricated, false or incorrect responses is a persistent limitation. This limitation undermines their reliability and remains a critical challenge, especially in areas where high precision and trustworthiness are required. To address this challenge, we investigate whether the features derived from Knowledge Graphs (KGs) align with the accuracy of answers produced by the LLMs. In particular, we focus on entropy-based KG features, which capture diversity and uncertainty within structured knowledge. By analyzing the correlation between the entropy-based KG features and the accuracy of LLM responses, we are able to identify “blind spots” where LLMs are prone to hallucination. This provides insights not only into when an LLM is correct, but also into the conditions under which it fails. We present results across several datasets, including two developed for this study, demonstrating that entropy-based KG features can effectively align with the accuracy of LLM responses. Motivated by these findings, we propose a probing strategy for assessing LLM accuracy by focusing on areas where LLM accuracy is weak. The experimental results confirm that KG features can guide the probing effectively, highlighting the importance of using structured features from KGs in building more reliable and hallucination-free AI based systems.