What question did this study set out to answer?

To develop an efficient fine-tuning strategy for vision-language models that mitigates issues caused by quantization.

May 7, 2026Open Access

QuantFT-VL: Harmonizing Quantization and LoRA for Efficient Mobile Vision–Language Model Fine-Tuning

FJFangyuan JinDalian University of Technology HLHui LinAnhui University LZLu ZhangDalian University of Technology

Key Points

To develop an efficient fine-tuning strategy for vision-language models that mitigates issues caused by quantization.
Introduced QuantFT-VL initialization after quantization for LoRA adaptation.
Used truncated singular value decomposition on quantization residuals for initialization.
Evaluated on six standard benchmarks comparing with QLoRA and full-precision LoRA.
QuantFT-VL improved average benchmark score by 3.27 percentage points over QLoRA.
Performance recovered to levels of full-precision LoRA in optimal settings.
Maintained memory and speed benefits of quantized fine-tuning.

Abstract

Vision–language models (VLMs) are increasingly deployed in resource-constrained environments, yet efficient fine-tuning remains challenging because post-training quantization often degrades the effectiveness of low-rank adaptation. This paper revisits that mismatch in the context of MobileVLM1.7B and presents QuantFT-VL, a novel initialization strategy following the quantization phase to seamlessly align with the LoRA technique. The key idea is to initialize LoRA using a low-rank approximation of the quantization residual instead of the default zero-initialization used in QLoRA-style pipelines. After quantizing a pretrained weight matrix W into Q, we compute the residual W − Q and use truncated singular value decomposition to initialize the LoRA factors (A and B) so that the starting adapted weight Q + ABT better matches the full-precision model. This residual-aware initialization reduces the discrepancy introduced by quantization and leads to faster and more stable optimization. Experiments on six standard VLM benchmarks show that QuantFT-VL consistently improves over QLoRA and recovers performance close to or better than full-precision LoRA in the best setting. On two RTX 3090 GPUs, QuantFT-VL improves the average benchmark score by 3.27 percentage points over QLoRA while preserving the memory and speed advantages of quantized fine-tuning.

Ask AI

Helpful

Bookmark

View Full Paper