This work presents a controlled empirical study of how internal representations scale in modern vision architectures, focusing on convolutional networks (ResNet, DenseNet) and Vision Transformers. Rather than relying on task-level performance metrics, we analyze final-layer representations using simple scalar probes: ℓ2 norm (magnitude), variance (dispersion), and effective rank (dimensionality). To isolate representation-level effects, all backbones are frozen and evaluated under two controlled ablation axes: model size and dataset size. Our findings show that representation behavior varies significantly across architectures and metrics. In particular, convolutional networks tend to expand effective dimensionality with increasing scale, while Vision Transformers exhibit increasing concentration into lower-dimensional subspaces. These results demonstrate that no single scalar metric sufficiently captures representation scaling behavior, highlighting the need for multi-metric analysis when studying internal representations beyond task performance.
Navya Chilaka (Sun,) studied this question.