Why Grok Groks: The 4/7 Threshold - A Geometric Theory of Delayed Generalization in Neural Networks

Key Points

• This research aims to explain grokking as a phase transition in neural networks at a threshold of 4/7.
• Developed a geometric theory of grokking
• Validated theory on NVIDIA H100 with 10/10 seeds
• Measured mean grok step with statistical analysis
• Achieved perfect generalization at Φ* = 4/7
• Mean grok step was 6690 with a standard deviation of 1273
• Establishes a connection between neural learning and Byzantine fault tolerance

Abstract