Why Self-Arbitration Looks Like Guessing: A Diagnostic Study of Small Language Models

When a 7–8B language model must choose between two conflicting solutions it generated itself, accuracy measured in a single presentation order looks close to guessing (about 45–64% correct). We study this on four models (Qwen2.5-7B-Instruct, Llama-3.1-8B-Instruct, Mistral-7B-Instruct-v0.3, and a self-correction fine-tune) and four math datasets (ASDiv, MATH levels 1–3, GSM8K, SVAMP), using a deterministic answer checker instead of an LLM judge. In the preregistered endpoint, a position-controlled skill index is above zero in all four endpoint cells, and also in the remaining 16 descriptive cells: the skill is real but hidden by the measurement. In exploratory analyses, the shortfall is mostly not position bias (at most 2.5 points) but the absence of a valid choice: only 51–90% of judgments select either solution, the rest being answer-format errors, repetition loops, re-solving, or an invented third answer. For Qwen this separation is not fully clean. Averaging probabilities over both orders helps only slightly and inconsistently. At the system level, using the model’s verdict loses to 3-sample majority voting in 15 of 16 cells, and even a perfect choice between two candidates beats 5-sample voting by at most 1.6 points.