What type of study is this?

This is a Quantitative Study study.

October 8, 2025Open Access

Infinite-Width Limit of a Single Attention Layer: Analysis via Tensor Programs

View Full Paper

Ask AI

Bookmark

Authors

This analysis identifies the infinite-width limit distribution in attention layers, revealing non-Gaussian properties.

The study reveals that the infinite-width limit distribution of a single attention layer shows non-Gaussian behavior, indicating fundamental departures from Gaussianity.
Using Tensor Programs, the exact form of the limit law is derived without reliance on infinite-head approximations or specialized scaling methods.
Numerical experiments confirm the theoretical predictions, demonstrating effective modeling of finite-width attention layers.
This work provides important insights for developing a unified theory of deep Transformer architectures in the infinite-width limit.

Sakai et al. (2025) studied this question.

View Full Paper

Ask AI

Bookmark