Evaluating large language models for automated design structure matrix extraction from unstructured documents: an empirical study

Design Structure Matrices (DSMs) capture dependencies between system entities and help analyze system complexity, but manually creating them from unstructured documents is time consuming. This work proposes an automated DSM extraction framework using LLMs and RAG with an explicit reasoning step before the LLM determines the presence of a dependency between two system entities. Using a hand-curated dataset, we evaluate three LLM models (GPT-4o-mini, GPT-3.5, and GPT-4o) across six performance metrics and cost.The findings show that reasoning length affects LLM’s DSM extraction performance.