CtrlAdapt: an adaptive multi-control framework for text-to-image diffusion models

Abstract Diffusion models now deliver state-of-the-art image synthesis, but vanilla Stable Diffusion offers limited spatial control. ControlNet injects structure via single-condition maps, leaving multi-condition training underexplored. In this work, we introduce CtrlAdapt, an adaptive multi-control framework for diffusion models that learns to dynamically fuse multiple control signals during image generation. CtrlAdapt extends ControlNet with a lightweight trainable adaptive fusion module that predicts data-dependent, spatially adaptive fusion weights over learned control features for multiple conditions. The combined conditioning is injected at all feature levels in the decoder, enabling adaptive modulation of conditioning information. Our approach uses semantic segmentation and depth maps and trains the adaptive fusion module jointly with the control branches while keeping the base diffusion model fixed. On the Cityscapes dataset, the proposed adaptive fusion achieves a FID of 65.96, compared to 67.95 for fixed-weight dual control and 69.11–70.79 for single-control variants. On the UrbanSyn dataset, adaptive fusion attains a FID of 70.38, substantially lower than the 90.99–96.09 range observed for single-control models. Across both datasets, the proposed method maintains competitive structural fidelity, with PSNR, SSIM, and LPIPS values comparable to fixed-weight fusion approaches.