Low-Rank Adaptation with a Single Matrix

Low-rank adaptation (LoRA) has become a cornerstone of parameter-efficient fine-tuning for large pretrained models, augmenting frozen pretrained weights with a low-rank update formed by the product of two smaller matrices. Recent research indicates that scale mismatches between these two factors frequently induce unstable training dynamics, leading to suboptimal performance. Here, we reformulate low-rank adaptation by parameterizing the weight update as a single low-rank matrix multiplied by its own transpose. This formulation, which we term SingLoRA , eliminates inter-matrix scale conflicts by construction, thereby promoting stable optimization, while approximately halving the number of trainable parameters. Through the lens of the efficient feature learning framework, we show that SingLoRA is provably stable by design while retaining the expressive capacity of LoRA within the transformer architecture. Extensive experiments on multiple tasks validate these benefits. Extensive experiments on multiple tasks validate these benefits. In common sense reasoning, fine-tuning LLaMA-7B on MNLI with SingLoRA achieves 91.3% accuracy—surpassing LoRA (89.1%) and LoRA+ (90.2%)—while using only 60% of their parameter budget. In image generation, fine-tuning Stable Diffusion with SingLoRA significantly improves image fidelity on DreamBooth, achieving a DINO similarity score of 0.151, compared to scores of 0.148 and 0.143 for DoRA and LoRA, respectively.