What type of study is this?

This is a Quantitative Study study.

October 10, 2025Open Access

On the Limitations and Capabilities of Position Embeddings for Length Generalization

Puntos clave

Position embeddings influence length generalization performance without expanding computational capabilities, but structuring learned computations.
The introduction of linear representation complexity characterizes when position embeddings enable length generalization effectively.
Empirical evidence supports the conjecture that sequential representation complexity must remain invariant across scales for length generalization.
Practical strategies, including scale hint and a learning-based position embedding framework, are proposed to improve length generalization.

Resumen

In Transformers, Position Embeddings (PEs) significantly influence Length Generalization (LG) performance, yet their fundamental role remains unclear. In this work, we investigate the limitations and capabilities of PEs in achieving LG. We theoretically analyze PEs in Position-Only Linear Attentions (POLAs), introducing Linear Representation Complexity (LRC) to characterize when PEs enable LG. Our analysis shows that PEs do not expand computational capabilities but structure learned computations across positions. Extending to practical Transformers, we propose Sequential Representation Complexity (SRC) and conjecture that LG is possible if and only if SRC remains invariant across scales. We support this hypothesis with empirical evidence in various reasoning tasks. To enhance LG, we introduce Scale Hint, allowing flexible instance scaling, and a Learning-Based Position Embedding framework that automatically learns positional relations. Our work provides theoretical insights and practical strategies for improving LG in Transformers.

Leer artículo completoexternamente

Preguntar a la IA

Me gusta

Guardar

Ver artículo completo