Answers are grounded in this paper and cite their sources.
Large language model (LLM) systems deployed in production are subject to a form of degradation that classical software is not: their behavior can change without a corresponding code change, because the underlying model, its provider-side weights, or the prompt scaffolding around it can shift independently of the application's own release cycle. Traditional regression testing, built around deterministic assertions, is poorly suited to this setting, because LLM outputs are open-ended, stochastic, and evaluated along dimensions –– tone, faithfulness, safety, format compliance –– that resist exact-match comparison. This paper presents LLM Maintainer, an open-source tool that treats a second LLM call as a judge, scoring a fixed suite of "golden prompts" against natural-language rubrics on every run, persisting the results, and flagging statistically meaningful drops relative to a rolling per-test-case baseline. We describe the system's architecture –– a YAML-configured test suite, an execution runner, a structured LLM-as-judge module, a regression-detection algorithm, a SQLite persistence layer, and a Flask dashboard –– and its command-line and dashboard-triggered workflows for both ad hoc and scheduled (cron-style) use. Because this work was produced in a network-restricted sandbox without access to the live Anthropic API, we report a fully disclosed, seeded, synthetic case study rather than presenting simulated data as genuine field results: we ran the actual, unmodified runner, judge, regression-detection, and persistence code for 30 simulated daily evaluation cycles (180 evaluated prompt-response pairs) against a six-case suite modeling a retail-banking customer-support assistant, with three deliberately injected incidents –– a transient API failure, a silent quality regression introduced by a system-prompt refactor, and a one-off output-format violation. The tool's naive, fixed-threshold rolling-baseline detector recovered five of six injected anomalies (83.3% recall) but also raised 18 flags attributable to ordinary sampling variance in a five-point ordinal judge scale (21.7% precision, F1 = 0.345), a quantitative finding that motivates statistically smoothed alternatives. We situate this design relative to LLM-as-a-judge evaluation research, holistic benchmarking, documented behavioral drift in commercial LLMs, and concept-drift detection in classical machine learning operations, and we discuss the threats to validity inherent in judge-based scoring –– including self-preference and verbosity bias –– alongside concrete mitigations. We conclude that continuous, judge-in-the-loop regression testing is a necessary complement to pre-deployment benchmarking for maintaining LLM-integrated software, while cautioning that naive thresholding requires the same statistical care long established in the concept-drift literature.
No takes yet. Share an insight, caveat, or question.
Ook Lee¹ (2026) studied this question.
Synapse has enriched 8 closely related papers on similar clinical questions. Consider them for comparative context: