Answers are grounded in this paper and cite their sources.
Selecting between visual-only and language-informed adaptation is a moving target imaging informatics decision. The same scarce target labels train both the candidate strategies and provide evidence for choosing between them. We introduce a fixed budget, policy value framework for evaluating whether this choice is worthwhile. Four medical vision-language models (BiomedCLIP, PubMedCLIP, MedCLIP, and MedSigLIP) were evaluated across chest radiography and seven MedMNIST tasks at 1–16 shots per class, with external multilabel validation on NIH ChestX-ray14. We compared visual and language-anchored probes and evaluated label-free, support-informed, cross-validated, and held-out meta-selection rules. Policy value was defined as the macro-AUROC gain over always using the visual probe at the same annotation cost. At one shot, language anchoring was superior in 31% of encoder–task combinations, yielding an optimistic oracle opportunity of at most 0.026 macro-AUROC. No deployable selector demonstrated a practically meaningful positive held-out value. Selector value was non-positive at one shot and at most + 0.002 at four shots, with aggregate hierarchical confidence intervals including zero, with the external test demonstrating the same pattern. Harmful language choices were more frequent and larger than beneficial choices, so selection accuracy did not translate into policy value. Controlled class description interventions changed the sign and magnitude of the language effect, demonstrating a genuine yet encoder- and task-dependent opportunity. Under the evaluated fixed budgets, direct visual adaptation was the most dependable use of target labels. Imaging informatics evaluations should report policy value, negative transfer severity, and annotation cost, rather than only selection accuracy.
No takes yet. Share an insight, caveat, or question.
Liatsis et al. (2026) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: