Scalable Physics-Informed Multi-Agent Reinforcement Learning for Building Energy System Control

Optimizing multi-zone building heating, ventilation, and air conditioning systems for energy efficiency while maintaining thermal comfort is a critical challenge, as buildings account for 40% of total energy consumption in the United States and approximately 30% globally. Although multi-agent reinforcement learning has emerged for building system control, existing studies typically involve fewer than 10 agents, rely on model-free methods despite having access to environment models, and lack real-building validation. This paper proposes a physics-informed model-based multi-agent reinforcement learning framework for scalable multi-zone control. First, we develop a physics-consistent graph neural network that combines group-shared multi-scale causal convolutions with a heat diffusion graph layer for inter-zone thermal coupling, enabling scalable multi-zone temperature prediction. Second, we introduce κ -neighborhood truncation for the multi-agent soft actor-critic algorithm, where each agent's critic receives only states within its κ -hop neighborhood, reducing critic input dimensionality by up to 72% with provable approximation guarantees. Third, we integrate the learned dynamics model as a differentiable world model for policy optimization, substantially improving sample efficiency over model-free alternatives. We validate the framework through a 114-day simulation study on 6 zones and a 42-day real-building deployment on 18 zones across 4 floors. The dynamics model achieves below 1.4°C mean absolute error across all zones. Ablation studies confirm that model-based κ-truncated training converges faster and to higher reward than model-free counterparts. The deployed controller achieves 15.7% and 35–70% energy savings in simulation and real-building studies, respectively, with modest thermal comfort degradation.