Projected-Feature Scale Calibration for 2-Bit Vision Encoders in Multimodal Large Language Models

Very low-bit quantization of a multimodal large language model’s vision encoder can impair text recognition and chart understanding. We present projected-feature scale calibration (PSC), an image-only post-training method for 2-bit vision encoders. PSC starts from range-searched GPTQ, fixes integer codes and zero points, and jointly updates existing group scales by matching features through a frozen multimodal projector. This objective aligns image-only calibration with the visual representation supplied to the language model. The learned updates fold directly into the original scales, preserving the inference architecture. Experiments on LLaVA-OneVision show consistent improvements in text recognition and chart understanding over the same strong quantized initialization, including under simultaneous language-model compression. Matched controls support projected-feature supervision over encoder-output reconstruction, repeated calibration samples confirm stable gains, and an unchanged-recipe contrastive language–image pre-training (CLIP) evaluation supports transfer to a second encoder. PSC thus improves the accuracy of compact vision encoders while preserving their quantized representation.