When the System Changes Underneath You

Production question-answering systems change without telling anyone: the model behind an API gets updated, a prompt gets tuned, the document corpus gets a new release. We show that the provenance layer we built for evaluation doubles as a runtime monitor: seeded probe questions with provable content guarantees, scored by exact counts with no judge model, detected a silent generator swap the day it happened, for less than a dollar per monitoring cycle. A full audit of the persisted replies then refined the diagnosis: the replacement model had not started fabricating — it had abandoned the refusal-sentinel contract, against a corrected baseline of exactly zero cross-document false answers in every unswapped cycle. The same coordinates survived two corpus releases (registered content migrated across document edits with 298/298 agreement against independently derived expectations) and a change of the monitored pipeline itself. Detection, diagnosis, and the audit connecting them rest on one property: every reply persisted, every verdict re-derivable. Companion to When the Scoreboard Lies (doi:10.5281/zenodo.22016705). The techniques described — content-guaranteed canary probes and provenance-registered span migration — are published as prior art; no patent applications are pending or planned. The artifact release follows at https://github.com/shamiksaharcciit-oss/provenance-canary.