Parameter-Efficient Fine-Tuning via General Linear Structural Regularization for High-Rank Adaptation

Parameter-efficient fine-tuning (PEFT) enables large language models to adapt to downstream tasks with low computational cost. As a representative high-rank PEFT method, MoRA (High-Rank Updating for Parameter-Efficient Fine-Tuning) improves update expressiveness through a compression–transformation–decompression reparameterization mechanism. However, its bottleneck subspace is still modeled using a freely learned linear transformation. In addition, grouped compression may project information from different original directions into shared bottleneck coordinates. This may reduce subspace separability and lead to inefficient utilization of the effective update space. To address this limitation, we propose GL-log-MoRA, which introduces a learnable general linear transformation into the MoRA bottleneck subspace and applies log-determinant regularization to encourage a more balanced spectral structure. In this way, the proposed method improves directional coordination and subspace expressiveness without imposing hard structural constraints or causing noticeable memory overhead. We evaluate GL-log-MoRA on five benchmarks: LogiQA, Financial PhraseBank, GSM8K, FinQA, and HotpotQA. The results show that GL-log-MoRA achieves the best performance on these downstream tasks and yields small but consistent improvements over MoRA under the same parameter budget. Compared with MoRA, GL-log-MoRA improves LogiQA from 42.50% to 45.45% and Financial PhraseBank from 81.60% to 83.02%. It also improves GSM8K from 63.1% to 64.6%, FinQA from 10.02% to 10.23%, and HotpotQA from 70.6% to 70.8%. Meanwhile, the average empirical effective-rank indicator increases from 1.05 to 2.80. Peak GPU memory changes only slightly, from 18.21 GB to 18.28 GB.