Modular post-hoc enhancements for zero-shot histopathology classification using vision–language models

Zero-shot learning (ZSL) with vision–language models (VLMs) has shown promising results in histopathological image classification. However, existing approaches often under-utilize domain-specific adaptations that could enhance performance in specialized medical tasks. In this study, we systematically evaluate three modular, post-hoc enhancements to standard zero-shot VLM pipelines: (1) Cosine Similarity Calibration for affinity matrix construction, (2) Prompt Template Enhancement via diverse clinical prompts, and (3) Adaptive Hyperparameter Tuning using a Gaussian Mixture Model (GMM)-based method for adjusting pseudo-labeling weights (λ) and support set contributions (γ), following Zanella et al. We conduct extensive experiments on 40,000 samples of histopathology benchmark using five VLMs: CLIP, CONCH, PLIP, Quilt-B16, and Quilt-B32. Results show that each enhancement module yields significant accuracy gains. Cosine calibration improves CLIP (+7.8%), CONCH (+1.53%), PLIP (+14.3%), Quilt-B16 (+16.1%), and Quilt-B32 (+11.4%). Prompt enhancement yields accuracy gains in PLIP (+16.8%) and Quilt-B32 (+29.9%), with limited effect on CLIP or CONCH. Adaptive tuning yields large improvements in PLIP (+28.4%). The best performance is achieved with the full combination of enhancements, with PLIP reaching 87.88% (+27.88%, p < 0.001) and Quilt-B32 reaching 88.38% (+36.1%) over their baselines. CLIP showed significance only with cosine calibration (p < 0.001), reflecting its contrastive learning backbone. Although CONCH reached the highest raw accuracy (92.63%) under the full enhancement setting, this performance appears primarily driven by its histopathology- specific pretraining; statistically significant gains emerged only in cosine-based combinations, indicating limited responsiveness to prompt or tuning. Our findings highlight the critical role of structured modular enhancements in optimizing VLM performance for specialized clinical domains.