Despite the limited empirical evidence, the potential for artificial intelligence (AI) large language models (LLMs) or chatbots to assist in structural engineering tasks has recently attracted much attention. To bridge this knowledge gap, this study systematically examines whether commonly used chatbots can reliably handle routine and complex steel analysis and design calculations, as well as building code compliance checks. Specifically, we evaluate six models—OpenAI’s o1 and GPT-4o, Anthropic’s Claude 3 and 3.5, and Google’s Gemini 1.5 and 2—via a cross-examination with a curated set of problems sourced from undergraduate and graduate textbooks as well as Principles and Practice of Engineering (PE) exam materials. Our methodology involves quantitative scoring on problem-solving accuracy, along with qualitative assessments of solution clarity, code references, and engineering judgment. Our findings provide empirical insights into each chatbot’s capacity to interpret structural engineering tasks, follow relevant steel design codes, and propose viable solutions. More specifically, newer models seem to outperform legacy models, most chatbots follow a structured approach to tackling analysis and design problems, and one of the persistent issues noticed is that most chatbots are not fully updated with the latest tabulated data and provisions in the current building codes and suffer from mathematical (calculation) slipups—which implies that with some tuning, chatbots are likely to be capable of conducting complete analysis and design procedures.
Eslamlou et al. (2026) studied this question.