ABSTRACT: Large Language Models (LLMs) have advanced the extraction and generation of engineering design (ED) knowledge from textual data. However, assessing their accuracy in ED tasks remains challenging due to the lack of benchmark datasets specifically designed for ED applications. To address this, the study examines how theoretical concepts from Axiomatic Design Theory—such as Functional Requirements, Design Parameters, and their relationship—are expressed in natural language and develops a systematic approach for annotating ED concepts in text. It introduces a novel dataset of 6,000 patent sentences, annotated by domain experts. Annotation performance is assessed using inter-annotator agreement metrics, providing insights into the challenges of identifying ED concepts in text. The findings aim to support designers in better integrating design theories within LLMs for extracting ED knowledge.
Giordano et al. (Fri,) studied this question.