Prompt-Robustness Tests Mostly Measure Format Compliance: Evidence from 48,000 Responses in a Question-Paired Design of Twelve Language Models

Prompt-perturbation tests rank language models by how well they cope with reworded, reformatted or padded questions. We find that for smaller models these tests mostly measure something else: whether the model ends its answer in the requested format. Twelve models answered 800 ARC-Challenge and GSM8K questions under five prompt conditions in a staged, pre-specified design. The seven strongest models barely moved under any condition. The five weaker models swung by up to 33 percentage points, but almost all of that swing came from format compliance. When unparsed answers were re-read with a pre-specified lenient rule, eight of the eleven significant condition effects disappeared and one reversed sign. The same confound can affect model selection: within Pilot 2, a stability-rewarding rule trailed plain leaderboard selection by 2.8 percentage points (3.5 in a separately seeded rerun), whereas the rule frozen from Pilot 1 and transferred once to Pilot 2 led by 0.25 points. These distinct evaluations do not establish a general advantage for either policy. We recommend that robustness studies report format compliance and accuracy on parsed answers as separate quantities. The evidence is pilot-scale and limited to two benchmarks.