Consensus-Guided Multi-Match Adaptation With Uncertainty-Calibrated Thresholding for Test-Time Cross-Modal Retrieval

This study enhances the robustness of pre-trained image–text retrieval models at test time, recovering the pre-trained similarity geometry under distribution shifts and modality degradations while preserving fine-grained cross-modal alignment. Existing Test-Time Adaptation methods, though effective for classification, are ill-suited to retrieval, where decisions require nuanced, bidirectional correspondences rather than single-label predictions. We address this gap with a framework that integratesConsensus-Guided Multi-Match Adaptation (CGMA)andUncertainty-Calibrated Matchwise Thresholding (UCMT)underBias-Only Fine-tuning. CGMA replaces single-candidate supervision with a consensus-guided strategy: it forms a temperature-scaled similarity distribution, selects Top-kmatches as pseudo-labels, filters queries by average confidence via percentile ranking, and aggregates the retained candidates into a consensus target, against which a bidirectional cross-entropy/Kullback–Leibler objective is optimized to mitigate modality corruption and local semantic interference. UCMT complements CGMA by treating similarity scores as evidential signals to derive a bounded, sample-specific uncertainty that sets a per-sample threshold for a matchwise penalty: predictions above it are promoted and those below suppressed, with the threshold rising for uncertain samples and falling for confident ones, thereby slowing gradients and preventing over-optimization. The two modules are jointly optimized with minimal overhead via bias-only updates, yielding state-of-the-art test-time performance under distribution shifts.