How Much Must Disappear? A Dose-Response Probe of Grounded Abstention in VLMs

How much of the visual evidence supporting an answer must disappear before a vision-language model stops answering? This paper frames grounded abstention as a dose-response problem rather than a single test at complete evidence removal. Starting from VQA items a model answers correctly, the probe progressively blackens the COCO-annotated answer object at six calibrated doses, measured as the fraction of object pixels removed, and estimates the object coverage at which explicit abstentions or valid absence reports reach 50%. At complete removal, the Grounded Abstention Index (GAI), a capability-conditioned Youden contrast, compares answer-object removal with a same-image control edit, separating evidence sensitivity from a general tendency to abstain after any edit. Across three open VLMs, behavior is strongly model-dependent: on a common 130-item support, Qwen2.5-VL-3B and Qwen2.5-VL-7B cross the 50% threshold at 0.81 and 0.76 object coverage, while Gemma-3-4B never does, and its GAI (0.30) is about half that of the Qwen models. The gap persists at matched int8 precision. Painted area alone does not explain the response: target edits that paint no more of the image than the control still raise Qwen grounded responses by 0.37–0.44. Prompt wording trades capability for restraint: permitting abstention raises Qwen-3B’s GAI from 0.23 to 0.56 but lowers its original-image accuracy from 0.82 to 0.59. Post-training quantization of Qwen-3B produces no degradation: int4 is indistinguishable from bf16, and int8 abstains slightly earlier, a shift detected by a logistic estimator butnot by interpolation. Code, manifests, and all model outputs are released.