Answers are grounded in this paper and cite their sources.
Low-rank adaptation (LoRA) is widely used for parameter-efficient fine-tuning, yet uniform rank assignment ignores the varying contribution of layers to a downstream task. Adaptive methods such as AdaLoRA allocate ranks via gradient and singular-value importance, but they do not exploit the representation change induced by the task. We introduce FLowRA, a calibration-based rank-allocation framework that combines a lightweight flow-change descriptor (velocity, acceleration, and directional alignment of hidden representations) with gradient activity to solve a constrained integer-budget problem. FLowRA complements existing LoRA variants, requires only two user-configurable parameters, and supports multiple architectures through architecture-specific implementations. On GLUE, SuperGLUE, instruction-following, code, vision, and vision-language benchmarks, FLowRA consistently outperforms uniform LoRA, DoRA, and AdaLoRA under low trainable-parameter budgets, reducing the gap to full fine-tuning by 48.5% on GLUE relative to the strongest baseline. We compare FLowRA with AdaLoRA using sensitivity-score correlations and report component ablations to assess the contribution of flow-guided allocation.
No takes yet. Share an insight, caveat, or question.
Samarin et al. (2026) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: