Abstract Parameter-efficient fine-tuning methods have shown promise for continual learning with pre-trained models, yet existing approaches either sacrifice performance or incur linear parameter growth with task count. We introduce Shared-A Low-Rank Adaptation (SA-LoRA), an asymmetric parameter-sharing strategy motivated by the observation that down-projection matrices tend to capture transferable, task-invariant features, whereas up-projection matrices retain task-specific diversity. SA-LoRA shares a single down-projection matrix across all tasks while maintaining task-specific up-projections. To enable flexible task balancing, we introduce separate normalization with learnable per-task scaling factors that decouple magnitude from direction. We evaluate SA-LoRA on three class-incremental learning benchmarks using Vision Transformer backbones. On ImageNet-R with 20 sequential tasks, SA-LoRA achieves 75.70% average accuracy with only 0.20M trainable parameters, outperforming parameter-efficient baselines. On the adversarial ImageNet-A benchmark, it reaches 57.22%, exceeding the strongest baseline by 4.89 points while using substantially fewer parameters than symmetric methods. These results demonstrate that asymmetric parameter sharing provides an effective balance between parameter efficiency and performance in continual learning.
Bing et al. (Mon,) studied this question.