Scaling Laws for Transformers on Low-Dimensional Data: A Statistical and Approximation Theory Perspective

Abstract: Scaling laws have emerged as a fundamental principle in characterizing the performance of transformer architectures across domains, revealing power-law relationships between model size, data availability, and predictive accuracy. While these laws are well-established in high-dimensional, data-rich environments, their validity in low-dimensional settings remains underexplored. This study examines transformer scalability on low-dimensional data through a dual lens of statistical learning theory and approximation theory, thereby establishing a principled framework that extends beyond empirical heuristics. The research problem addressed is the lack of theoretical grounding for scaling laws in constrained feature spaces, where over-parameterization and limited sample regimes alter approximation behaviors. Methodologically, the study integrates theoretical derivations with controlled empirical evaluations on representative low-dimensional datasets, enabling comparison between predicted and observed scaling dynamics. Results indicate that conventional power-law behaviors deviate under low-dimensional constraints, with approximation-theoretic limits exerting dominant influence on generalization. The findings propose reformulated scaling laws that capture these dynamics, demonstrating superior predictive alignment with empirical performance. The impact of this study lies in its contribution to a unified theoretical framework of transformer scalability, informing efficient model design in resource-constrained domains such as econometrics, biomedical analysis, and scientific computing. By bridging statistical theory and approximation analysis, this research advances both theoretical understanding and practical applicability of scaling laws across diverse data regimes. Keywords transformer architectures, scaling laws, low-dimensional data, statistical learning theory, approximation theory, generalization, model complexity, power-law behavior, constrained datasets, empirical validation