Experimental evaluation reveals convergence on standard syntax trees over novel parsing architectures across independent AI sessions, indicating limits in architectural invention from requirements...
Could an AI have invented the CCS parsing model from the requirements it satisfies? Five fresh Claude Code sessions, each with no access to the project, were given one 350-word brief stating seven requirements (five-language consumption of one parse output, byte-for-byte regeneration, a consumer that works from the output alone, diff-based cross-language agreement, self-hosting, no semantic actions, LL(1) with reading as the hot path) and asked for an architecture-level design. A sixth fresh session scored the five blind. All five converged on the same structure: a flat, lossless concrete syntax tree laid over the verbatim input, in the tree-sitter and Roslyn lineage. None produced the four-entity (Context, Name, Symbol, Rule) model that replaces the tree in CCS, and none produced its two-phase architecture of a mutable runtime followed by a bidirectional binary. Of the six published departures from the YACC lineage, three were forced by the requirements, one only weakly, and two not at all. The paper records the protocol, the five designs, the blind scores, the informed comparison, an argument for why behavioral requirements cannot carry an ontological choice, the experiment's known weaknesses, and two follow-up rounds. > If you can program a little, you can follow this. The question is > simple: give an AI a list of requirements, and does it come back > with the invention those requirements were written from? We > measured it. The answer is no, and the interesting part is why.
No takes yet. Share an insight, caveat, or question.
Aleksandr F. Urakhchin (2026) studied this question.
Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context: