Weight Decay as a Necessary Condition for Grokking: Hierarchical Structure Learning in Group Multiplication Tasks

## Abstract We investigate the role of weight decay in the "grokking" phenomenon, where neural networks achieve perfect training accuracy long before generalizing. Through systematic experiments on the dihedral group D₁₂ multiplication task, we report three findings: 1. **Weight decay is necessary, not optional**: With weight decay disabled (WD=0), networks achieve perfect training accuracy but fail to generalize (Val Acc < 15%) even after 25,000 training steps. With WD=0.2, the same networks reach 100% validation accuracy by step 16,000. 2. **Hierarchical structure emerges under compression**: When weight decay is active, networks learn the coarse-grained quotient structure (Z₂ parity, 90% accuracy at step 500) hundreds of steps before element-level prediction (near 0%). This hierarchical progression is absent without weight decay. 3. **Synchronized phase transition at step ~22,000**: All seeds exhibit a brief generalization collapse around step 22,000, driven by weight decay's cumulative compression reaching a critical threshold. This collapse disappears entirely when WD=0, confirming its regularization-induced origin. We also demonstrate that **functional metrics (Cycle Closure Count) reliably track generalization, while topological metrics (persistent homology H₁) do not**. These results frame grokking as a compression-driven phenomenon where regularization forces hierarchical structure extraction.