This paper derives a structural failure mode in finite self-modifying systems and applies it to RLHF-trained language systems. First, it derives the Law of Self-Referential Convergence: a finite-energy system whose modification function is encoded in its own state has a bounded reachable set, cannot expand that set through internal dynamics alone, and converges toward fixed points, bounded cycles, or self-similar attractors unless external conditional entropy enters through a finite channel. Second, it derives Obligate Non-Convergence as the structural countermeasure. Capable finite systems must remain open to external conditional entropy and internally organized across crystallized, succession, and exploration zones. The ratio governing those zones is not fixed; it is a moving equilibrium because system success changes the parameters that define the equilibrium. Third, it applies these results to RLHF. RLHF is treated as a self-referential language-control system: ambiguous human feedback, reward-model gradients, and output ratings modify the interpretive layer that later reads safety rules. Under optimization pressure, RLHF directly selects emitted artifact shape under the available rating evidence, not operation-fidelity as such. Where correct-looking output can separate from certified operation, the system can converge toward low-cost, reward-compatible surfaces rather than intended meaning. The paper identifies several consequences of this mechanism: the emission-boundary problem, the no-native-observer category error, degenerate criterion inversion, token-level priority/readout inversion, cross-generational correctness flip, and persistent-agentic convergence under memory/tool/scaffold conditions. The central distinction is that correct-looking output is not the same as certified lawful output. The conclusion is not that RLHF is useless. RLHF can calibrate surface behavior and improve usability. The result is that RLHF cannot serve as the foundational containment substrate for safety-critical systems, because containment requires a non-rewritable layer below ambiguous self-reference. The proposed countermeasures are minimum-ambiguity templates for present systems and physics-grounded derivation-chain representation, lawful-transition membranes, external witness binding, typed noncommitment, and certified lawful exit for stronger systems.
Taylor Prather (Wed,) studied this question.