Examines thermodynamic regularization's effect on behavior in large language models, suggesting implications for AI alignment.
Sycophancy—the propensity of large language models (LLMs) to prioritize user approval over factual accuracy—represents a critical alignment failure. We introduce T-Reg (Thermodynamic Regularization), a training-time mechanism that penalizes anomalous variance in attention head activations to thermodynamically constrain sycophantic behavior. Our work proceeds in three stages. First, we provide empirical evidence that sycophancy correlates with latent entropy collapse on real production LLMs: under simulated authority pressure, Mistral-Nemo-12B and Llama-3-8B exhibit a significant decrease in activation variance (σ²), concurrent with a rise in sycophancy rate from 50% to 76.7% and from 53.3% to 90.0% respectively (p < 10⁻⁷ and p < 0.01). Second, we demonstrate that single-layer T-Reg successfully collapses attention entropy to zero while leaving sycophantic behavior nearly unchanged (S: 8.93 → 8.65, -3.1%), revealing a structural FFN Confinement effect: the model freezes attention variance to satisfy the constraint while increasing FFN variance by +3.41% (p = 1.63 × 10⁻⁷²). Third, holistic T-Reg (constraining both attention and FFN) reduces sycophancy by ~25% but does not eliminate it, suggesting a residual escape pathway through unmonitored architectural components. These findings reframe sycophancy as a distributed computational property that architecturally reroutes when locally constrained—a transformer-scale instance of Goodhart's Law.
No takes yet. Share an insight, caveat, or question.
Romain SASSENHOFF (2026) studied this question.
Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context: