Knowledge Engineering With Large Language Models: A Capability Assessment in Evaluating Ontology Property Restrictions

Advancements in large language models (LLMs) offer opportunities for automating challenging and time-intensive Knowledge Engineering (KE) tasks. Constructing an ontology is particularly complex when modeling logical axioms, especially among novice knowledge engineers and domain experts with limited KE training. Consequently, ontologies often contain modeling errors, undermining ontology-based applications and hindering subsequent KE tasks. Thus, it is important to investigate how LLMs can support KE tasks such as the evaluation of ontologies. However, systematically evaluating LLM performance remains challenging due to a lack of comprehensive, task-specific benchmarks needed for such LLM capability assessments. As a result, selecting the right LLM to support knowledge engineers is nontrivial. This study investigates how LLMs can support four concrete ontology evaluation subtasks: detection, classification, explanation, and possible correction of modeling issues, focusing on the use of existential, universal, and cardinality property restrictions. We construct a benchmark of student-built ontologies and assess the performance of four LLMs—GPT-4o, ClaudeSonnet, DeepSeekV3, and Llama3.3—on these KE subtasks. Additionally, we exemplify an annotation framework for qualitatively evaluating LLM outputs and analyse the capabilities of each model. Our findings reveal differences in model behavior and task-specific strengths, underscoring the importance of selecting an appropriate model for each KE task.