This analysis reveals the limits of maintaining safety and utility in self-modifying systems, suggesting critical implications for future designs.
Can a safety gate permit unbounded beneficial self-modification while maintaining bounded cumulative risk? We formalize this question through dual conditions — requiring Σδₙ < ∞ (bounded risk) and ΣTPR_n = ∞ (unbounded utility) — and establish a theory of their (in)compatibility. We prove classification impossibility: for power-law risk schedules δₙ = O(n⁻ᵖ) with p > 1, any classifier-based gate under overlapping safe/unsafe distributions forces ΣTPR_n < ∞ via Hölder's inequality (Theorem 1). This is exponent-optimal (Theorem 3), and a second proof via the NP counting method yields a 13% tighter bound (Theorem 4). A universal finite-horizon ceiling (Theorem 5) shows that for any summable risk schedule, maximum classifier utility grows subpolynomially — orders of magnitude below a verifier's linear growth. A Lipschitz ball verifier escapes the impossibility entirely, achieving δ = 0 with TPR > 0 (Theorem 2), validated on GPT-2 (d_LoRA = 147,456). Comprehensive empirical validation is in the companion paper.
No takes yet. Share an insight, caveat, or question.
Arsenios Scrivens (2026) studied this question.
Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context: