Answers are grounded in this paper and cite their sources.
Disagreement across paraphrases flagged wrong answers better than disagreement across resamples at the same call budget, but both signals are weak because most errors are confident. On DeepSeek-chat with 1,600 MMLU and 1,600 TriviaQA questions, AUROC for detecting errors with 3 extra answers was 0.695 (paraphrase) vs 0.646 (resample) on MMLU (difference +0.049, 95% CI 0.011 to 0.089) and 0.785 vs 0.668 on TriviaQA (+0.117, CI 0.065 to 0.169). Both differences remain significant after Holm correction. About 70% of errors are confident (at least 4 of 5 samples agree on the same wrong answer). Resampling flags 10% of them and paraphrasing 40%, at a higher false-alarm rate (9.4% vs 4.0%). Caveats: one model, DeepSeek writes and checks its own paraphrases, the MMLU gap shrinks to +0.031 (CI -0.009 to 0.071) when questions with missing paraphrases are kept, and 27% to 30% of audited "errors" were label noise or ambiguous questions. These are preliminary results. Preliminary results. Single model (DeepSeek-chat), 1,600 MMLU and 1,600 TriviaQA questions. Hand audits were first drafted with an LLM (Claude) and entered by the author; the report text was written with LLM assistance.
No takes yet. Share an insight, caveat, or question.
Prerna Sachdeva (2026) studied this question.
Synapse has enriched 8 closely related papers on similar clinical questions. Consider them for comparative context: