Answers are grounded in this paper and cite their sources.
Building energy modeling (BEM) is a key instrument for assessing energy demand and CO 2 emissions of the building stock and for planning decarbonization strategies. Its reliability, however, critically depends on accurate geometric and material input data, including facade-related parameters such as window areas and window-to-wall ratios that strongly affect solar gains and transmission losses. These facade components can be extracted from images, but current approaches are dominated by supervised deep learning models, which require large annotated datasets, generalize poorly across cities and architectural styles, and struggle with real-world occlusions from vegetation and street furniture. To address these limitations, we propose a zero-shot semantic segmentation workflow with integrated occlusion compensation. The segmentation stage is based on a vision-language model (VLM), and the occlusion compensation stage uses a generative image editing model. Both stages are controlled purely via text prompts and do not require any task-specific fine-tuning. We systematically analyze prompt designs and evaluate the workflow across six facade datasets, including a newly introduced Lemgo Musicians’ Quarter (LMQ) dataset of 92 detached and multi-family residential buildings with pixel-accurate annotations. The experiments show that a unified vision-language-mask model (SAM 3) combined with a flow-matching generative model (FLUX.1 Kontext) achieves the best overall performance: generative occlusion compensation improves Mean IoU on LMQ from 0.760 to 0.801. The downstream analysis shows that occlusion compensation is particularly beneficial for building-area estimation, while its effect on WWR is smaller but removes a systematic overestimation; minor window hallucinations remain a relevant limitation. In cross-dataset evaluation SAM 3 achieves the highest Mean IoU of 0.602, outperforming the strongest per-dataset-trained supervised baseline by approximately 8.6 percentage points, and remaining competitive even against supervised baselines trained on the combined data of all six datasets, while performing on par in intra-dataset settings. By enabling accurate, scalable extraction of facade components with minimal reliance on human-annotated training data, the proposed approach helps mitigate a key data bottleneck in BEM and supports more transferable, up-to-date assessments of building energy performance.
No takes yet. Share an insight, caveat, or question.
Sprute et al. (2026) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: