Reading the Board, Playing Another One: Turn-Level Evaluation of Open LLM Agents on Textual Sokoban

Aggregate outcomes hide where an interactive agent fails. We study five open 3-8B LLM agents on textual Sokoban: 180 episodes and 1,561 steps over six conditions, in which no level is solved. Two language-model annotators code each turn against a behavioural taxonomy induced from three blind reading passes. A push-optimal replay through the repaired harness solves every level, and no model solves any of the 60 ladder episodes that need four or more pushes, where random play almost always fails. Two observations are supported, at different strengths. Across the five models, the move a thought describes and the action it emits differ in kind, usually a plain move described and a push emitted: both annotators label this in 111 of 180 episodes (kappa 0.749); in one 32B model, one annotator labels it on none of 172 pushes. The prompt's only worked example has this shape, and a pre-registered arm that made it consistent left its effect unresolved. In the one model that writes its own board table into its thought, an accurate table, probably copied, sits beside prose that reasons from a different board (14 episodes on the reading passes; 6 to 27 of 180 on the annotations). A pre-registered positive control, where 12.76 of 20 solves are chance-expected, finds no separation between solved and unsolved episodes, so the taxonomy is a description, not a validated discriminator.