CubeAttn: Training Dynamics, Kernel Failure Modes, and Seed Sensitivity in Linear Attention

We present a systematic empirical study of a linear-complexity attention operator (CubeAttn B+) that replaces the cross-token attention matrix with a lightweight global aggregation mechanism. Our investigation spans four rounds of experiments with increasing rigor, including 275+ new trained models and three position-encoding interventions. We report findings with methodological implications: (1) a U-shaped kernel failure curve shows intermediate convolution kernels (k=5-11) cause systematic collapse while k=3 and k=15 perform well; (2) 5-seed validation reveals single-seed results are fundamentally unreliable, with Long-Range Recall overestimated by up to 50%; (3) a diagnostic-driven design cycle testing RoPE-based and per-layer absolute position encoding interventions converges on a single architectural constraint: the V-aggregation + feature-dim softmax design averages away per-position information, making absolute position binding structurally impossible without changes to the aggregation mechanism.