Exact Machine Unlearning in Non-Convex Attention Networks via Stiefel Manifold Projection of A-NTK Downdates

Machine unlearning—the task of perfectly erasing the influence of specific training data from a learned model—has been rigorously solved for strongly convex objectives via Newton-step downdates. However, extending exact unlearning to Large Language Models (LLMs) remains an open problem due to the non-convex loss landscape induced by softmax attention, layer normalization, and multiplicative gating. All existing LLM unlearning methods are strictly approximate: they bound the Kullback–Leibler divergence by some epsilon > 0, leaving cryptographic traces detectable by white-box Membership Inference Attacks (MIAs). We present a novel theoretical framework that achieves exact unlearning for finite-width transformer architectures. Our approach consists of three key contributions: 1. Attention-Neural Tangent Kernel (A-NTK): We define a layer-recursive kernel that maps the non-convex attention weight space into a strictly convex Reproducing Kernel Hilbert Space (RKHS), enabling closed-form algebraic downdates in the infinite-width limit. 2. Stiefel Manifold Projection: We prove that projecting this infinite-width downdate back to finite-width networks via a Cayley retraction on the Stiefel manifold bounds the KL divergence to O(1/sqrt(h)), which vanishes as width h approaches overparameterization. 3. Tractable Computation: We achieve computational scalability via layer-wise Kronecker-Factored Approximate Curvature (K-FAC), reducing Hessian inversion from O(d^3) to O(sum l d_l^3). Extensive experiments on LLaMA-3 8B demonstrate that our method achieves 100% forget efficacy, 98.3% retain utility, and MIA success rates indistinguishable from random guessing (AUC = 0.50), while requiring only 52 GPU-hours—a 269x speedup over full retraining.