Efficient encoder fine-tuning through configuration search and sparse low-rank adaptation

Low-Rank Adaptation (LoRA) reduces the task-specific optimization burden of transformer fine-tuning, but its effectiveness remains sensitive to rank, placement, scaling, dropout, and training choices. This study examines whether automated configuration search provides a consistent advantage when tuning effort is controlled. Genetic Algorithm (GA), Slime Mould Algorithm (SMA), Optuna, and Pareto-GA were evaluated in a shared mixed search space under a common 25-candidate validation budget, followed by five-seed retraining of selected configurations. DistilBERT experiments covered Emotion, AG News, and Financial PhraseBank, with RoBERTa-Base/SST-2 used as a secondary cross-model check. No optimizer consistently outperformed tuned references. On AG News, full fine-tuning and GA+SU-LoRA reached 94.4 ± 0.2% and 94.3 ± 0.2% accuracy. On Financial PhraseBank, the corresponding values were 86.0 ± 1.0% and 86.2 ± 0.9%, a descriptive difference within the observed retraining variability. Emotion ablations showed that SU-only was numerically comparable to GA+SU-LoRA, whereas Sparse-only and GA+Sparse LoRA differed by 0.1 percentage point. For GA+Sparse LoRA, fixed unstructured masks suppressed an average of 51.6% of LoRA-factor coefficients; the dense trainable allocation remained approximately 0.706M because tensor shapes were unchanged, and merged dense-kernel inference showed no latency benefit. Pareto selection exposed a clearer validation performance-support trade-off than the original scalar compactness penalty. The results emphasize matched baseline tuning, explicit component controls, analytically consistent parameter accounting, and deployment-aware evaluation.