Answers are grounded in this paper and cite their sources.
Tool-using agents are routinely improved by fine-tuning on their own successful trajectories, and “successful” almost always means that the final answer was correct. But a correct answer need not be justified by what the agent observed. On a closed-world ProofWriter tool-use task, a sound checker certifies that the observed evidence determines the answer in only about one in seven of a 4B agent’s correct trajectories, and 316 of 877 outcome-selected training trajectories submit before observing anything. Fine-tuning the agent on them barely helps: the student often answers without looking and scores below an agent that gives the same answer to every question. We instead keep a correct trajectory only if a deterministic evidence-sufficiency certificate, which replays the agent’s tool calls, shows that the observed replies determine the answer under the declared semantics. With the same model, trajectory source, episode count and update count, certificate selection lifts macro accuracy from 39.7% to 59.3% (+19.6 points, 95% CI [18.4, 20.8], positive in all five seeds; our pre-specified primary contrast) and beats a length control, an observation-ratio heuristic and an LLM judge; against a judge told the task’s semantics the comparison is inconclusive. The advantage replicates in smaller and larger students of the same family (+22.4 and +28.7 points) and, more weakly, in a second model family (+4.4 points, 95% CI [3.0, 5.8]). Students remain weak on question types their teacher never certified, and all results come from one synthetic benchmark.
No takes yet. Share an insight, caveat, or question.
Song Xu (2026) studied this question.
Synapse has enriched 8 closely related papers on similar clinical questions. Consider them for comparative context: