What question did this study set out to answer?

This study aims to explore the training dynamics and performance issues in CubeAttn, a linear attention model.

June 9, 20260 citationsOpen Access

CubeAttn: Training Dynamics, Kernel Failure Modes, and Seed Sensitivity in Linear Attention

Key Points

This study aims to explore the training dynamics and performance issues in CubeAttn, a linear attention model.
Conducted four rounds of experiments with 275+ trained models.
Examined three types of position encoding interventions.
Analyzed performance based on kernel sizes ranging from k=3 to k=15.
Identified a U-shaped kernel failure curve indicating collapse at intermediate kernels (k=5-11).
Found that single-seed validations can overestimate performance metrics, particularly Long-Range Recall, by up to 50%.
Determined that the V-aggregation and feature-dim softmax design hinder absolute position information due to aggregation mechanics.

Abstract

We present a systematic empirical study of a linear-complexity attention operator (CubeAttn B+) that replaces the cross-token attention matrix with a lightweight global aggregation mechanism. Our investigation spans four rounds of experiments with increasing rigor, including 275+ new trained models and three position-encoding interventions. We report findings with methodological implications: (1) a U-shaped kernel failure curve shows intermediate convolution kernels (k=5-11) cause systematic collapse while k=3 and k=15 perform well; (2) 5-seed validation reveals single-seed results are fundamentally unreliable, with Long-Range Recall overestimated by up to 50%; (3) a diagnostic-driven design cycle testing RoPE-based and per-layer absolute position encoding interventions converges on a single architectural constraint: the V-aggregation + feature-dim softmax design averages away per-position information, making absolute position binding structurally impossible without changes to the aggregation mechanism.

问 AI

Bookmark

View Full Paper