Demonstrates enhanced reranking in scientific code generation, indicating improved accuracy without prior training.
When a language model writes scientific code, the result usually runs and returns a plausible number that can still be wrong by a sign, a unit, a factor, or a violated conservation law — and the model's own confidence does not tell you which. We study verification grounded independently in the underlying science: a declarative correctness checker, built from a public schema-validated knowledge corpus spanning physics, chemistry, biology, climate, mathematics, and numerical methods, and never trained on preference labels. Applied at sampling time as a best-of-N reranker over an un-tuned base model — an approach we call verifier-guided sampling — it supplies, without preference labels or training, the reranking signal that learned reward models (RLHF, DPO, RLVR) are used to provide. On a 73-prompt scientific-code benchmark and two open-weights model families (Llama 3.1 8B and Mistral Nemo 12B), the deployable verification-only signal selects the same candidate as a functional ground-truth oracle on 97.3% of prompts; across an N-sweep (N in {3, 5, 10, 20, 50}) it raises the mean overall score by up to +0.108 (Llama) and +0.148 (Mistral Nemo) and captures at least 95% of the available rerank headroom at N=50, robustly across vendors. The contributions are (i) an information-theoretic analysis of what any post-hoc verifier can recover from a generation distribution, separating an outlier-rejection regime from a partial-credit regime; (ii) a sample-budget sweep characterising the diminishing-returns curve; and (iii) sampling-policy ablations (fixed-N, adaptive-N, and beam-with-verifier). A direct comparison against a trained reward model is left to dedicated follow-up work. The benchmark, the run records, and the open Lemma verification substrate are publicly released.
No takes yet. Share an insight, caveat, or question.
Arsalan Akhtar (2026) studied this question.
Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context: