Dynamic Modality Reweighting for Robust Vision–Language Models under Noisy Multimodal Environments

Recent Vision Language Models (VLMs) perform poorly when either the visual or textual part is affected by noise like blur, occlusion, or unclear text. This paper presents a Dynamic Modality Reweighting (DMR) framework that balances the contributions of visual and textual features based on their estimated reliability during inference. The structure includes a Confidence Estimation Network (CEN) to evaluate trust scores for each modality, followed by a Dynamic Fusion Layer (DFL) that combines embeddings using data-driven weights. Experimental results on noisy versions of the MS-COCO, Flickr30k, and Visual Genome datasets show up to 23% improvement in multimodal consistency and a 17% decrease in semantic drift when compared to baseline CLIP and BLIP-2 models.