Assessing ChatGPT's Capability in Extracting Architectural Elements from Software Systems

Software architecture plays a critical role in ensuring system quality, yet manually identifying architectural patterns and tactics in codebases remains time-consuming and error prone. While traditional tools like Archie automate parts of this process, the emergence of large language models (LLMs) like ChatGPT offers new opportunities for scalable, intelligent analysis. This work aims to investigate the capability of ChatGPT, a large language model (LLM), for software design analysis, specifically in detecting architectural patterns and tactics. We evaluated ChatGPT-4o and Archie on five open-source systems (Apache Storm, Flink, Spark, Gradle, and Maven) using precision, recall, F1-score, and accuracy metrics. The methodology involved: (1) processing entire codebases (via zipped uploads) and architecturally significant snippets, (2) manually validating outputs against a ground truth, and (3) comparing results with Archie’s established benchmarks. Results reveal that ChatGPT achieves 57% accuracy for pattern detection outperforming Archie (44.4%) in systems like Spark and Gradle, where modern patterns (e.g., Pipeline, MasterSlave) are prevalent. However, it achieves a precision of 23% vs. Archie’s 75% for tactic detection, highlighting limitations in recognizing traditional architectural strategies. Both tools face challenges with false positives, but ChatGPT demonstrates superior adaptability to newer paradigms. These results suggest that LLMs can augment—but not yet replace— traditional tools for architectural analysis. ChatGPT’s strength in pattern extraction bridges gaps in automation, while its tactic detection limitations underscore the need for hybrid approaches. This study provides empirical insights into LLMs’ role in software engineering, paving the way for more intelligent, collaborative architecture analysis tools.