Domain Pretraining Buys the Ranking, Not the Scale: A Calibration Audit of Medical Vision-Language Models

This paper audits the calibration quality of a domain-pretrained medical vision-language model against a general-domain one, asking whether medical pretraining buys better-calibrated confidence or only better discrimination. Two frozen ViT-B/16 backbones, BiomedCLIP (pretrained on fifteen million biomedical figure-caption pairs) and OpenAI CLIP, are compared on two closed-set classification tasks derived from PMC-VQA (imaging modality and anatomy) under zero-shot similarity and a linear probe, before and after temperature scaling, across five seeds. All headline metrics are computed on the human-verified evaluation subset, and confidence is assessed under four metrics (ECE under two binning schemes, MCE, Brier score, and negative log-likelihood) so that disagreement between them is visible rather than resolved by selection. The dissociation is complete. The domain-pretrained model is the more accurate model in all four experimental cells and the worse-calibrated model in all four before scaling. The mechanism generalises past these two artefacts: a contrastive objective constrains which class wins and never by how much, so the same feature separability that lifts accuracy inflates the maximum probability zero-shot and, by admitting a heavier regularisation penalty under an accuracy-selected sweep, deflates it under a probe. The regime of use, not the backbone, sets the sign of the confidence error, an inversion with opposite failure modes at clinical triage. Temperature scaling reverses the calibration ranking in three cells of four, but only where the fitting slice resembles the evaluation slice. Bounded by 285 and 295 evaluation items, the study defends a direction and a mechanism, not a magnitude: a model selected on accuracy carries no warrant of trustworthy confidence.