Answers are grounded in this paper and cite their sources.
A team comparing two related versions within a model family on a few-shot benchmark can pick a different winner just because of which worked examples, the demonstration set, precede the test questions. Some setups use one fixed set for every question and model. We scored 29 versions from six model families on 14 tasks: nearby pretraining checkpoints, successive post-training stages and successive releases. Every version answered the same fixed questions under the same 64 randomly drawn sets. A pre-specified test with family-wise error control confirmed the problem in five of six families: in 30 comparisons of related versions within a family, both accuracies were in a prespecified range and the average over the 64 sets had a clear direction, yet the less frequent direction appeared in at least 19 of the 64 sets. The remedies are exploratory. With one evaluation pass, rotate sets across questions (Li et al., 2025). To decide which version is better, average several full runs, each with its own set; Section 8 gives model-based planning estimates of how many.
No takes yet. Share an insight, caveat, or question.
Ariel Elboim (2026) studied this question.
Synapse has enriched 8 closely related papers on similar clinical questions. Consider them for comparative context: