Key points are not available for this paper at this time.
Large language models (LLMs) have exhibited remarkable capabilities across diverse open-domain tasks, yet their application in specialised domains such as civil engineering remains largely unexplored. This paper starts bridging this gap by evaluating and enhancing the reliability and robustness of LLMs in structural analysis of beams. Reliability is assessed through the accuracy of correct outputs under repetitive runs of the same analysis problem, whereas robustness is evaluated based on the performance across varying load and boundary conditions. A benchmark dataset, comprising eight beam structural analysis problems, is created to test the Llama-3.3 70B Instruct model. Results show that, despite an apparent qualitative understanding of structural mechanics, the LLM lacks the quantitative reliability and robustness required for engineering applications. To address these limitations, a shift is proposed that reframes the structural analysis as a code generation task. Accordingly, an LLM-empowered agent is developed that (a) integrates chain-of-thought and few-shot prompting to generate accurate OpenSeesPy code, and (b) automatically executes the code to produce structural analysis results. Experimental results demonstrate that the agent achieves accuracy exceeding 99.0% on the benchmark dataset, exhibiting reliable and robust performance across diverse conditions. Ablation studies highlight the complete example and function usage examples as the primary contributors to the agent’s enhanced performance.
Liu et al. (Thu,) studied this question.