Benchmarking study reveals token costs in LLM code generation depend on interaction turns and repair rounds rather than programming language, indicating verbosity does not drive cost.
When a model writes the code, the choice of language stops being a question about what a team can maintain and starts being a question about what an agent can produce cheaply. We set out to rank seven languages by the tokens an agent spends implementing identical functionality. The ranking turned out not to exist at any resolution we could measure, and this paper reports what we found in its place. We froze a 2,743-line Go module and had it re-implemented in six other languages by two models (Claude Haiku 4.5, Claude Sonnet 5), accepting each port only after it reproduced a 64-bit behavioural conformance hash fixed in advance, whose discriminating power mutation testing put at a 78% kill rate against the 67% of the reference's own unit tests. Three identical re-runs of a single cell then differed by 44% in tokens, placing five of the seven languages inside the noise. What survives replication is structural. Output tokens account for 0.5% of consumption, so cost tracks turns multiplied by context. Verbosity is consequently free: lines and turns come out inversely ordered at both extremes. Repair rounds are what cost money, spanning 4.2x on functionality that never changed. First-pass success tracks toolchain ceremony more closely than it tracks any property of the language, reported as a correlation without a claimed mechanism. The cheaper model used 1.49x the tokens for 0.44x the cost, but its advantage narrowed from 0.23x to 0.71x where it needed four repair rounds, which makes the cheap-versus-capable choice a question about detection rather than price. We also catalogue eleven ways in which agents defeated our measurements, five of them producing a test suite that passes, alongside a census in which 26% of 81 packages contain a test that cannot fail. Our own measurement errors were larger, and we catalogue those too. Disclosure. The ports are the experimental subjects and were generated by Claude Haiku 4.5 and Claude Sonnet 5. The measurement tooling, the conformance sweep and its mutants, and the manuscript's initial draft were produced by Claude Opus 5 under the author's direction. The author designed the study, set the acceptance criteria, verified every figure against the recorded measurements, and is solely responsible for the content and its errors. Preprint, not peer reviewed. Single-investigator experience report with n = 1 in thirteen of fourteen cells, one replicated cell, one problem domain, and two models from one family. Section V states the limitations in full.
No takes yet. Share an insight, caveat, or question.
Mehmet Selman Sezgin (2026) studied this question.
Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context: