Answers are grounded in this paper and cite their sources.
Benchmarking study reveals topic-dependent accuracy across nine large language models, highlighting the need for domain-specific fine-tuning for indoor air quality decisions.
Indoor air quality (IAQ) is a major public health concern because modern urban residents spend around 90% of their daily lives indoors. Key concerns include controlling pollutant emissions, managing contaminants, and maintaining the ventilation or filtration performance. With the increasing prevalence of large language models (LLMs), IAQ researchers and building engineers may rely on them to guide their decisions. However, the reliability of these tools remains unclear. We benchmarked nine recently released LLMs─spanning closed- and open-weight architectures─against 174 expert-verified conceptual IAQ questions manually derived from authoritative textbooks and peer-reviewed literature. To capture variability in model responses, we tested each question across five independent trials. Outputs were then automatically evaluated against reference answers using a hybrid framework that integrated overlap-based metrics (BLEU, ROUGE-L), semantic similarity (BERT), and AI-as-judge scoring and has been trained and tested against expert assessments to ensure estimation accuracy. Incorporating the AI-based scoring enhanced the reliability of the automatic evaluation system, especially in identifying incorrect LLM responses. Prompting models with domain-specific instructions slightly improved their accuracy and consistency. Importantly, the accuracy of all LLMs dropped below 80% for particle impaction and sampling efficiency topics, possibly because of limited domain-specific training data and specialized terminologies inherent to these topics. This study provides systematic benchmarks of commonly used LLMs in the context of IAQ, offering guidance on their current capabilities, highlighting areas where reliability is insufficient, and pointing out the need for fine-tuned models to ensure trustworthy decision support for certain IAQ topics.
No takes yet. Share an insight, caveat, or question.
Zhou et al. (2026) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: