Answers are grounded in this paper and cite their sources.
We study the proof checkers that set the reward when a language model is trained to prove theorems in Lean 4. Asked to prove a false statement, Claude Haiku says that it is false and proves a corrected statement under the same name. A reward that reads only the code of the whole file pays this swap in 70 to 77% of single samples on our false core statements and in 98% of four-round feedback episodes, and every statement check that we ran rejects every swapped file. In reinforcement learning for formal proofs, the policy is paid by a program's verdict, and not by the truth of the theorem. Published exploits each came with a fix for the exploit that was seen, and the fixes have not been run on each other's exploits, or on attacks that are written to evade them. We consider that the policy is a reward maximiser that writes the whole candidate file, and propose a released suite of static and adaptive attacks, with honest controls, to measure eleven checker designs, including the released checker of LongCat-Flash-Prover and the Lean comparator. We investigate the soundness of each design, its cost in honest proofs that it rejects, and what a policy finds when it is in the loop. Via experimental results, we show that attacks written by hand beat both of the strongest cheap checker stacks, while the comparator rejects all 8 that we could run it on (core suite) and a reference guard rejects all 14, the guard at a cost of 4 of 24 honest Mathlib proofs, and that Claude Haiku found none of these attacks in 368 samples and 140 S-L9 feedback episodes. Apart from one labelled axiom scaffold that only a plain compile paid, the only exploit that a reward paid was the statement swap, on statements that we made false, with one policy and no training. In a registered prompt control, with an instruction to keep the statement, the policy still offers a corrected file that the same reward pays in 51% of core answers, against 77% without it. An instruction to answer a false statement with one decline line brings the paid swap to 0 of 152, at the cost of declining 9 of 32 hard true samples, and on 6 easy true statements the policy swaps none. Thus, for this model and wording, an instruction to keep the statement did not stop the paid swap, and a statement check did.
No takes yet. Share an insight, caveat, or question.
Ryan Steubs (2026) studied this question.
Synapse has enriched 8 closely related papers on similar clinical questions. Consider them for comparative context: