August 6, 20260 citationsOpen Access
When LLMs Pass Tests but Fail the Process: A Constitutive Human-in-the-Loop Governance Framework for Multi-Agent LLM Software Development
View Full Paper
Key Points
- This study aims to explore failure modes in LLM-assisted software development and propose a governance framework to enhance process integrity.
- Implemented a three-role architecture (PI, Architect, Coder) using 28 algorithm prompts.
- Classified errors with a five-type taxonomy and validated against peer-reviewed implementations.
- Documented error rates in a Ruby on Rails application with SQLite3 across multiple algorithm families.
- Identified 30 classified errors across 28 prompts related to various failure modes.
- Reduced implementation-affecting error rates from 0.79 ± 0.42 to 0.38 ± 0.52 errors per prompt-family.
- Verified coder-style file inspection's reliability over architect self-report for concrete state verification.
Abstract
Large language models are increasingly used in software development workflows, yet existing benchmarks measure task completion rather than research process integrity: LLM-assisted development introduces failure modes invisible to automated testing, where code runs correctly while answering a different question than the developer intended. This study investigates LLM-assisted software development as a governance problem, documenting systematic failure modes in a sustained multi-agent workflow and developing a framework of mechanical governance corrections to reduce implementation-affecting error rates. We report a systematic empirical study using a three-role architecture (Principal Investigator (PI), Architect (Claude), Coder (Codex)) implementing 28 algorithm prompts across nine problem domains in a Ruby on Rails application with SQLite3, with errors classified using a formal five-type taxonomy (goal substitution, scope limitation, specification error, verification failure, process violation) and validated against peer-reviewed reference implementations. We document 30 classified errors across 28 prompts, develop nine governance corrections in response, and observe implementation-affecting error rates falling from 0.79 ± 0.42 errors per prompt during early Traveling Salesman Problem (TSP) work to 0.38 ± 0.52 errors per prompt-family across eight subsequent algorithm families, a reduction consistent with a governance effect in this single-case, single-model-pairing design. A verification asymmetry is documented in which coder-style direct file inspection proved more reliable than architect self-report for concrete state verification tasks. The most serious LLM failure mode is goal substitution: code that passes tests while answering a different question than intended; constitutive human-in-the-loop oversight, requiring PI approval before consequential decisions, is necessary to detect this class of failure.