From Parameters to Manifolds: Understanding the Limits of Scaling and the Geometry of Learning

Description This work proposes a shift in how we understand modern deep learning systems. Instead of seeing neural networks as collections of millions or billions of parameters, this paper argues that the true object of learning is the geometry and topology of the representation manifold—the hidden space where activations live, interact, and reorganize during training. The paper introduces a geometric framework in which training is interpreted as the continuous deformation of this manifold. Each gradient update subtly changes local distances, reshapes curvature, and alters the connectivity structure of the conceptual space. Under this view, many behaviors previously treated as unrelated phenomena emerge as natural geometric consequences. Hallucination appears as geodesic instability. Catastrophic forgetting becomes a topological collapse. Grokking corresponds to a sudden reconfiguration of global structure. Abstraction arises as the condensation of clusters and the opening of new shortcuts. A key contribution of the work is that it offers a practical method for analyzing these geometric properties without requiring any special training pipeline. Using only standard tools—Jacobian sensitivity, k-nearest-neighbor geodesic estimation, diffusion distances, and topological data analysis—researchers can study the manifold structure of existing pretrained models. This makes the framework accessible to anyone, even without large-scale computational resources. The study also confronts one of the most pressing questions in AI today: why scaling laws are beginning to show diminishing returns and unpredictable behavior. The argument made here is that parameter count alone cannot explain the stability or capability of advanced models. Geometry becomes the bottleneck. A model’s ability to generalize, remain aligned, or avoid pathological behavior depends on how well its internal manifold is cultivated, not merely on how large it is. Taken together, this work proposes a new paradigm: learning as manifold cultivation. It unifies insights from representation learning, information geometry, topology, and emergent behavior into a coherent picture of how intelligent systems grow. The resulting theory offers both an explanation for current limitations and a roadmap for the next generation of AI research.