The Promise and Challenge of Large Language Models for Knowledge Engineering: Insights from a Hackathon

Knowledge engineering (KE) is the process of building, maintaining and using knowledge-based systems. This recently takes the form of knowledge graphs (KGs). The advent of new technologies like Large Language Models (LLMs) has the potential to improve automation in KE work due to the richness of their training data and their performance at solving natural language processing tasks. We conducted a multiple-methods study exploring user opinions and needs regarding the use of LLMs in KE. We used ethnographic techniques to observe KE workers using LLMs to solve KE tasks during a hackathon, followed by interviews with some of the participants. This interim study found that despite LLMs' promising capabilities for efficient knowledge acquisition and requirements elicitation, their effective deployment requires an extended set of capabilities and training, particularly in prompting and understanding data. LLMs can be useful for simple quality assessment tasks, but in complex scenarios, the output is hard to control and evaluation may require novel approaches. With this study, we aim to evidence the interaction of KE stakeholders with LLMs, identify areas of potential, and understand the barriers to their effective use. We find copilot approaches may be valuable in developing processes where the human or a team of humans is assisted by generative AI.