Organizations face increasing challenges in managing the growing complexity of products and the associated data volume, especially in virtual product development. Maintaining competitiveness in dynamic markets necessitates shorter development cycles, which in turn depend on efficient access to relevant information without manual querying of internal data sources. This study investigates the integration of Large Language Models (LLMs) with enterprise-specific knowledge graphs to enable intuitive natural language querying. Specifically, two approaches for Cypher query generation are introduced and evaluated: one based on prompt engineering (KG-Prompt-LLM) and the other combining prompt engineering with vector embeddings (KG-Embed-LLM). Both approaches operate on a knowledge graph developed at the BMW Group, tailored to represent dependencies in the product and information flow in the design processes. A total of 235 validated user prompts, executed ten times each, were used to benchmark both approaches with respect to retrieval accuracy. Results show that while KG-Prompt-LLM provides a lightweight solution, its reliance on regular expressions leads to ambiguity in entity recognition. In contrast, the KG-Embed-LLM approach significantly improves precision through semantic matching. This work demonstrates the potential and current limitations of LLM-assisted knowledge graph querying in industrial contexts and provides a foundation for future research in LLM-based enterprise information systems.
Lehmacher et al. (Thu,) studied this question.