Influence of material attribute context and document layout variability on large language model performance in construction material specification

Construction material specifications are crucial for procurement, but their varied and unstructured presentation in PDF datasheets hinders traditional information extraction. Pre-trained large language models (LLMs) offer new opportunities to automate this process, through retrieval-augmented generation (RAG) with natural language-based instruction prompts. However, an unclear understanding of document characteristics in material specifications may hinder the LLM’s application in the construction material technical comparison process. This research aims to identify the influence of material attribute context and document layout variability on large language model performance in construction material specification extraction framework. The research method applied an LLM-based framework to automate the extraction of construction material specifications from PDF-based technical datasheets. The performance evaluations conducted across four different construction material types revealed the influence of document layout variability and material attributes on extraction accuracy. Attributes lacking clear identifiers and varying terminologies showed low performance (∼23.7%), whereas clearly defined terminologies achieved high accuracy (∼96.2%). Additionally, complex multi-column layouts hindered performance, causing a 30 percentage-point drop in accuracy compared to single-column formats. Among the models, Anthropic Claude 3 Haiku achieved the highest accuracy and showed strong robustness to complex layouts, highlighting the value of standardized, clearly labelled datasheets for improved machine-readability in the construction industry.