Knowing When Not to Answer: Evaluating Confidence and NLI-Based Verification for Selective Question Answering

Selective question answering allows a system to abstain from predictions that are likely to be incorrect rather than answering every input. This work investigates whether semantic verification provides a better basis for selective abstention than model confidence alone. Using a pretrained extractive QA model on SQuAD v2, the study compares temperature-scaled model confidence, question-aware semantic verification, answer-support NLI verification, and fixed confidence-verification fusions under a leakage-controlled held-out evaluation protocol. On the final held-out set of 3,000 examples, calibrated confidence achieves the best global AURC of 0.292379, compared with 0.309120 for confidence plus answer-support verification and 0.339417 for confidence plus question-aware verification. Paired bootstrap analysis and pairwise ranking diagnostics show that the verification signals are diagnostically informative but introduce more harmful than beneficial reordering when fused with confidence. The results support the narrower conclusion that additional semantic verification does not necessarily improve selective QA ranking over a strong confidence baseline under the evaluated models and fusion rules.