The Mirror Law: Reference Quality and the Transfer of Systematic Bias in Imitation and Distillation

When a model is trained to match a reference — a teacher network in knowledge distillation, or a reward model in RLHF — it never observes its own error against the ground truth. It observes only its disagreement with the reference. We make this elementary but consequential observation precise. For squared loss, the pointwise training residual decomposes exactly as the difference of two error fields, D(x) = eθ (x)−δ(x), where eθ is the learner’s deviation from truth and δ is the reference’s. The minimizer of the observed objective over an expressive hypothesis class therefore satisfies eθ = δ: the learner reproduces the reference’s error field (and, when under-expressive, only its representable component). Splitting δ into a systematic component (bias) and a zero-mean component (noise) yields three regimes with sharply different behavior. With a faithful reference the learner converges to truth; with a noisy reference the noise averages out and the learner still converges to truth (subject to a finite-data memorization caveat); with a biased reference the learner clones the bias — and, critically, the training loss falls toward zero while the true error plateaus at the bias magnitude, so the failure is undetectable from the optimization trajectory alone. We call this deceptive descent. We give an operational calibration criterion, borrowed from measurement science, for when a reference is informative and when a learner should stop trusting it, and we show it predicts graded references and a graduation point. We connect the resulting taxonomy to knowledge distillation, learning from noisy labels, reward-model over-optimization and sycophancy, and recursive model collapse. We run a minimal, fully reproducible experiment that exhibits all three regimes, the deceptive-descent signature, and the calibration relation, and confirm the effect survives nonlinearity; a classification experiment under a spurious correlation confirms that a teacher’s emergent bias transfers under distillation while symmetric label noise does not, and both a controlled proxy-reward optimization and a synthetic RLHF-style pipeline—a reward model learned from preference comparisons and optimized by PPO against a held-out gold evaluator—reproduce reward-model over-optimization, in which the measured reward inflates while true quality declines; and a recursive-distillation experiment confirms that model collapse contracts the noise component while systematic bias persists or accumulates. Finally, a real-LLM study — DPO post-training of Qwen3.5-0.8B and 2B with both fixed-reference and cloud-judge (RLAIF) arms — reproduces deceptive descent and the matched-error Regime-B-vs-C separation in an autoregressive model, and exhibits a predicted capacity-dependent robustness in which the larger model’s pretrained prior resists a verbosity-biased judge; an author-annotated human-feedback (RLHF) arm, in which an author supplies controlled-bias preference labels, reproduces the same pattern, with the smaller model collapsing under a human length bias while the larger partially resists. We are explicit throughout about what is classical and what is new: the estimator-level facts are known; the contributions are the unifying decomposition, the regime taxonomy, the trajectory-undetectability result, the calibration criterion, the non-realizable refinement, and the cross-domain mapping. We distill the results into a Mirror Audit Protocol for grading references, detecting deceptive descent, and deciding when to stop, graduate, or decorrelate supervision.