LLM-Based Knowledge Engineering for DSS Collaborative Knowledge Bases: Approach and Pilot Study

The creation of collaborative knowledge bases for decision support systems (DSS) mitigates the subjectivity of individual experts and enhances overall system efficacy. However, traditional knowledge engineering approaches are highly labor-intensive when eliciting and integrating expert knowledge, as they require extensive, expert-level interaction between knowledge engineers and domain specialists. Modern large language models (LLMs) and retrieval-augmented generation (RAG) technologies present novel opportunities for overcoming these limitations. This study presents a pilot investigation to assess the potential of LLM-based knowledge engineering for developing collaborative knowledge bases within knowledge-based DSS, thereby assisting decision-making in complex operational scenarios involving technical systems. The article proposes an LLM-based approach for creating collaborative knowledge bases, including extraction, consolidation of expert knowledge and evaluation of their operability. To implement and evaluate the proposed approach, specialized prompts were engineered, and pilot experiments were conducted to generate consolidated knowledge cases through expert-LLM interactions. The resulting knowledge cases were subsequently applied in an experimental decision-making inference procedure for fault diagnosis in gas-fired heating boilers. During this inference process, an LLM agent, guided by tailored prompts and a RAG-enabled knowledge base, interactively queries the user to identify the specific issue and subsequently proposes a contextually appropriate solution. Throughout this study, the LLMs demonstrated capabilities in dialogue management, expert knowledge elicitation, and knowledge consolidation, successfully facilitating the creation of a collaborative knowledge base grounded in the “Event-Cause-Symptoms-Action” model. The findings highlight the viability of future research in LLM-based knowledge engineering and support the further advancement of the “LLM-as-knowledge-engineer” paradigm.