An Empirical Evaluation of Large Language Models Applying Software Architectural Patterns

Beyond code generation, large language models (LLMs) are increasingly explored in software architectural tasks. However, it remains unclear to what extent LLMs can apply explicitly requested architectural patterns when provided with user-defined requirements. In this paper, we empirically evaluate the ability of multiple LLMs to generate specific architectural styles under controlled conditions. Models are prompted with specific requirements expressed in different ways and are instructed to generate architectures in four typical styles using the same single prompt and execution strategy. The authors assess the generated architectures with respect to specific evaluation criteria. The results show that, while LLMs can correctly apply simpler architectural patterns, performance decreases as architectural complexity and problem size increase. Model size and requirement representation appear to influence pattern adherence, whereas retrieval-augmented generation (RAG) produces mixed effects. The findings contribute empirical evidence regarding prompting strategies, requirement representations, RAG configurations, and diagram-as-code (DaC) representations for LLM-generated software architectures. The study also introduces a reusable experimental workflow intended to support future benchmarking and comparative evaluation in software architecture generation tasks.