Multi-Task Unified Domain Incremental Learning with Domain Difference Adapters

This paper focuses on domain incremental learning (DIL) for multiple vision tasks, including object detection, instance segmentation, and image classification. DIL aims to adapt a model to new domains over time without forgetting previously acquired knowledge. Recent DIL methods append learnable prompts to input embeddings of a frozen base model to learn from new domains. However, due to prompts' limited representation ability, they struggle to adapt the feature space to new domain data distributions. To overcome this limitation, we propose a novel DIL method named Domain Difference Adapters (DD-Adapters). Through feature visualization and singular value analysis, we identify the cross-domain clustering ability of the base model and the low-rank property of domain difference. Based on these insights, our method imposes low-rank constraints on the base model to capture the principal components of domain differences, while freezing the base model to maintain its cross-domain clustering ability, thereby adapting to new domains effectively. Additionally, we introduce a prototype-guided domain selector (PDS) to dynamically select the appropriate DD-Adapters during inference, mitigating catastrophic forgetting in DIL. Extensive experimental evaluations on eight benchmark datasets demonstrate the performance superiority of the proposed method on three vision tasks, with minimal extra parameter usage.