Answers are grounded in this paper and cite their sources.
Empirical evidence shows that fully-connected neural networks in the infinite-width limit (lazy training) eventually outperform their finite-width counterparts in most computer vision tasks; on the other hand, modern architectures with convolutional layers often achieve optimal performances in the finite-width regime. In this work, we present a theoretical framework that provides a rationale for these differences in one-hidden-layer networks; we derive an effective action in the so-called proportional limit for an architecture with one convolutional hidden layer and compare it with the result available for fully-connected networks. Remarkably, we identify a completely different form of kernel renormalization: whereas the kernel of the fully-connected architecture is just globally renormalized by a single scalar parameter, the convolutional kernel undergoes a local renormalization, meaning that the network can select the local components that will contribute to the final prediction in a data-dependent way. This finding highlights a simple mechanism for feature learning that can take place in overparametrized shallow convolutional neural networks, but not in shallow fully-connected architectures or in locally connected neural networks without weight sharing. Fully connected neural networks in the infinite-width limit often outperform finite-width models, while convolutional networks excel at finite widths. Here, the authors uncover how convolutional networks leverage local, data-dependent kernel renormalization, enabling feature learning to absent in fully connected architectures.
No takes yet. Share an insight, caveat, or question.
Aiudi et al. (2025) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: