Leveraging Graph Transformers for Knowledge Editing in Large Language Models

Large Language Models (LLMs) have transformed natural language processing, yet their static knowledge bases present significant challenges, such as temporal obsolescence and factual inconsistencies, limiting their effectiveness in real-world applications. Knowledge editing offers an efficient alternative to full-model retraining by enabling targeted updates to an LLM’s knowledge. However, existing approaches often overlook the ripple effects of edits on interconnected information, resulting in limited generalization. To address this, we propose a new model that leverages gRaph trAnsformers for Knowledge Editing in large Language models (RAKEL). RAKEL employs nucleus sampling to construct subgraphs that capture the cascading impacts of knowledge edits, preserving contextual relationships. A graph transformer then encodes these subgraphs, modeling both local and global dependencies within the knowledge graph to improve the representation of interrelated knowledge. Additionally, a constrained learning strategy ensures alignment between the edited model’s output distribution and that of the original model, reducing overfitting to updated knowledge. We evaluate RAKEL on two datasets, where it outperforms state-of-the-art baselines across multiple metrics. These results highlight RAKEL’s ability to deliver precise and generalizable knowledge updates, particularly in complex multi-hop reasoning scenarios.