Multi-agent validation is a natural defence against the unreliability of a single LLM judge: asecond, independent agent scores the same item, and disagreement routes the case to a human.But this design breaks down exactly when the system starts to learn. If the primary scoreradapts to human corrections through retrieval-augmented exemplars while the auditor stays acold skeptic, every freshly learned lesson re-creates the disagreement it was meant to resolve,and the system escalates precisely the cases it just learned to handle. We present Mizaan, aproduction-grade multi-agent scoring system, and three mechanisms that address this tension,evaluated across two contrasting domains (software-engineering work items and real-estatelistings) over three independent runs with no fine-tuning anywhere in the loop. Our central result is human-governed memory consolidation. Human corrections act first asinstance-level retrieved exemplars seen only by the primary; when corrections teaching the samelesson cluster in embedding space, an LLM drafts the lesson as a rubric amendment, a humanapproves it, both agents then inherit it, and the promoted exemplars are retired. We find thatinstance-level learning lifts similar held-out items above unrelated controls — a 0.20 separation(+0.33 vs. +0.12), which we take as the learning measure rather than the absolute lift — butleaks onto same-domain items it should not affect (+0.35); on the domain where correctionscluster, consolidation eliminates this leakage — returning controls to baseline while preservingthe learned lift — so the policy channel is measurably more precise than the instance channelthere. Second, inter-agent disagreement is a reproducible signal: the agents agree on 88.3 ± 0.5%of items across runs. Third, counterfactual cold scoring — scoring each item with and withoutretrieved corrections — serves as a routing signal, but its primary value at scale is as a diagnostic:it exposes a systematic auditor offset that warm-gap routing would mask, and we characterizeprecisely when inter-agent disagreement can be attributed to learning rather than error. Adeterministic spot-check canary surfaces deliberately injected bad lessons; we characterize thesample sizes a usable overturn rate would require. Mizaan’s inter-agent agreement replicatesacross two further base models including a cross-vendor one, and its learning separation stayspositive throughout, though its magnitude — like the absolute score shifts — varies with thebase model; these supplementary runs cover instance learning only, so consolidation robustnessacross base models remains future work. Throughout, learning is immediate, inspectable, andrequires no gradient update, GPU, or redeployment.
Mohammed Jawed (Mon,) studied this question.