Continuous Test-Time Adaptation aims to adapt a source model to continuously and dynamically changing target domains. However, previous studies focus on adapting to each target domain independently, treating them as isolated, while ignoring the interplay of interference and promotion between domains, which limits the model’s sustained capability, often causing it to become trapped in local optima. This study highlights this critical issue and identifies two key factors that limit the model’s sustained capability: (1) The update of parameters lacks constraints, where domain-sensitive parameters capture domain-specific knowledge, leading to unstable channel representations and interference from old domain knowledge and hindering the learning of domain-invariant knowledge. (2) The decision boundary lacks constraints, and distribution shifts, which carry significant domain-specific knowledge, cause features to become dispersed and prone to clustering near the decision boundary. This is particularly problematic during the early stages of domain shifts, where features are more likely to cross the boundary. To tackle the two challenges, we propose a Dual Constraints method: First, we constrain updates to domain-sensitive parameters by minimizing the representation changes in domain-sensitive channels, alleviating the interference among domain-specific knowledge and promoting the learning of domain-invariant knowledge. Second, we introduce a constrained virtual decision boundary, which forces features to move away from the original boundary, and with a virtual margin to prevent features from crossing the decision boundary due to domain-specific knowledge interference caused by domain shifts. Extensive benchmark experiments show our framework outperforms competing methods.
Song et al. (Tue,) studied this question.